2024-03-29T11:21:29.558 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:21:29.558 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716 --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/orig.config.yaml 2024-03-29T11:21:29.598 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:21:29.689 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622716' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsw3D26UudSyhWAXvfU35it+Nkk+lU6EapXFrTTx4yL3V7UtRqrKhirIsmYe1pOnNZZ/beEEY1Wztdgab8ahDU= smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQvsr5fyqktjrh12Z71e6Rxc5GaVazbKAgCg4Zym6vhpl4LSoM7JfyI6EhK4lynfYfgcQvapZ4K0zZRCmK1gb8= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:21:29.689 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:21:29.690 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:21:29.690 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:21:29.691 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:21:29.715 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:21:29.715 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:21:29.715 INFO:teuthology.packaging:ref: None 2024-03-29T11:21:29.715 INFO:teuthology.packaging:tag: None 2024-03-29T11:21:29.715 INFO:teuthology.packaging:branch: main 2024-03-29T11:21:29.715 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:21:29.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:21:29.999 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:21:30.000 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:21:30.008 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:21:30.009 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:21:30.018 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:21:30.032 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:21:30.040 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:21:30.061 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.256330', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsw3D26UudSyhWAXvfU35it+Nkk+lU6EapXFrTTx4yL3V7UtRqrKhirIsmYe1pOnNZZ/beEEY1Wztdgab8ahDU='} 2024-03-29T11:21:30.083 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.257805', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQvsr5fyqktjrh12Z71e6Rxc5GaVazbKAgCg4Zym6vhpl4LSoM7JfyI6EhK4lynfYfgcQvapZ4K0zZRCmK1gb8='} 2024-03-29T11:21:30.083 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:21:30.090 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:21:30.090 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:21:30.090 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:21:30.155 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa3a19f20d0>, signals=[15]) 2024-03-29T11:21:30.156 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:21:30.165 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:21:30.166 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:21:30.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:30.259 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:21:30.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:30.353 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:21:30.360 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-03-29T11:21:30.377 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-03-29T11:21:30.378 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-03-29T11:21:30.434 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:21:30.435 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:21:30.437 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-03-29T11:21:30.465 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-03-29T11:21:30.483 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-03-29T11:21:30.483 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-03-29T11:21:30.539 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:21:30.540 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:21:30.541 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-03-29T11:21:30.565 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:21:30.576 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:21:30.583 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:21:30.583 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:21:30.586 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:21:30.601 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:21:30.611 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:21:30.612 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:21:30.642 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:21:30.751 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:21:30.792 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:21:30.809 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:21:30.809 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:21:30.809 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:21:30.809 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:21:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:30.826 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-29T11:21:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:21:30.827 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-03-29T11:21:30.882 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:21:30.883 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-361.el9.x86_64 2024-03-29T11:21:30.883 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-03-29T11:21:30.886 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:21:30.886 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-361.el9.x86_64 2024-03-29T11:21:30.886 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-03-29T11:21:33.011 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:21:33.098 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:21:34.474 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T11:21:34.678 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-29T11:21:35.448 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:21:35.502 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-29T11:21:37.291 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-29T11:21:37.396 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T11:21:41.416 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.1 MB 00:00 2024-03-29T11:21:41.946 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-29T11:21:42.717 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 16 kB 00:00 2024-03-29T11:21:43.506 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-29T11:21:43.710 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-29T11:21:46.598 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-03-29T11:21:48.802 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-29T11:21:51.001 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:21:51.001 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:21:51.100 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:21:51.108 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:21:51.108 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:21:51.108 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:21:51.108 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:21:51.108 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-03-29T11:21:51.109 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:21:51.110 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 76 M 2024-03-29T11:21:51.110 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 123 M 2024-03-29T11:21:51.110 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:21:51.287 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:21:51.555 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T11:21:51.829 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:21:52.462 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 25 MB/s | 34 MB 00:01 2024-03-29T11:21:52.821 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T11:21:52.823 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:52.824 INFO:teuthology.orchestra.run.smithi081.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T11:21:52.824 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:21:53.174 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:21:53.174 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:21:53.829 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:21:53.829 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:21:53.910 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:21:53.923 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:21:53.924 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-03-29T11:21:53.925 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:21:53.925 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 76 M 2024-03-29T11:21:53.925 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 123 M 2024-03-29T11:21:53.925 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:21:54.052 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:21:54.052 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:21:54.082 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:21:54.965 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-29T11:21:55.440 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T11:21:55.757 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 16 MB/s | 28 MB 00:01 2024-03-29T11:21:55.761 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:55.761 INFO:teuthology.orchestra.run.smithi071.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-29T11:21:55.761 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:21:56.132 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:21:56.133 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:21:56.383 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:21:56.949 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:21:56.950 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:21:56.995 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:21:57.080 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:21:59.255 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:21:59.837 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:21:59.858 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:21:59.951 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:21:59.985 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:02.689 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:02.831 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:04.679 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:04.775 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:07.578 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:07.781 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:09.469 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:12.487 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:38.894 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:38.894 INFO:teuthology.orchestra.run.smithi081.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:22:38.894 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:22:38.937 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:39.300 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:39.300 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:39.300 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:39.300 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:22:39.703 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:22:39.703 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:22:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:39.704 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:22:39.704 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:22:40.004 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:40.105 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:40.106 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:40.106 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:22:40.106 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:22:40.106 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:22:40.106 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:22:40.106 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:22:40.106 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:40.186 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:40.186 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:40.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-03-29T11:22:40.221 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-03-29T11:22:40.222 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:22:40.223 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:22:40.223 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:22:40.223 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:22:40.223 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-03-29T11:22:40.223 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:22:40.223 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:22:40.224 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:22:40.224 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:22:40.224 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:22:40.857 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-03-29T11:22:42.022 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:42.022 INFO:teuthology.orchestra.run.smithi071.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:22:42.023 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:22:42.065 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:42.292 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:22:42.332 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-03-29T11:22:42.334 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-29T11:22:42.349 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.mWPHpM28bp 2024-03-29T11:22:42.349 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mWPHpM28bp 2024-03-29T11:22:42.408 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:42.408 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:22:42.408 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:22:42.408 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:22:42.418 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.mWPHpM28bp 2024-03-29T11:22:42.556 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.mWPHpM28bp is 6KB 2024-03-29T11:22:42.604 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.mWPHpM28bp 2024-03-29T11:22:42.618 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:22:42.691 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:22:42.691 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:22:42.691 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:22:42.691 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:22:42.692 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:22:42.851 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:22:42.851 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:22:42.851 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.851 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.851 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.852 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:22:42.852 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:22:42.852 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:22:42.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-03-29T11:22:43.179 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:43.280 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:43.280 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:43.280 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:22:43.280 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:22:43.280 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:22:43.280 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:22:43.280 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:22:43.281 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:22:43.360 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:22:43.360 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:22:43.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-03-29T11:22:43.402 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 68.el9 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8264254 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:22:43.403 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:22:43.404 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:22:43.405 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:22:43.405 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:22:43.405 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:22:44.023 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-03-29T11:22:45.311 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:22:45.363 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-03-29T11:22:45.365 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-29T11:22:45.380 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.TdwIix9IrT 2024-03-29T11:22:45.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TdwIix9IrT 2024-03-29T11:22:45.449 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.TdwIix9IrT 2024-03-29T11:22:45.588 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.TdwIix9IrT is 6KB 2024-03-29T11:22:45.637 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.TdwIix9IrT 2024-03-29T11:22:45.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:22:45.724 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:22:45.724 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:22:45.724 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:22:45.724 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:22:45.726 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:22:45.866 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-03-29T11:23:12.873 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:23:12.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-29T11:23:12.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:23:15.870 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:23:15.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T11:23:15.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:12.920 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:24:15.875 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:24:21.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-29T11:24:21.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:24.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T11:24:24.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:25.227 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-29T11:24:25.387 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-03-29T11:24:25.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-29T11:24:25.815 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:24:25.815 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-03-29T11:24:25.829 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:24:25.829 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:24:25.829 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:24:25.829 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:24:25.830 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T11:24:25.830 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:24:25.831 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-29T11:24:25.844 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:24:25.844 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:24:25.844 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:24:25.845 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:24:26.830 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:24:26.831 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:24:26.846 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:24:26.846 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:24:26.901 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-03-29T11:24:26.938 DEBUG:teuthology.parallel:result is None 2024-03-29T11:24:26.948 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-03-29T11:24:26.984 DEBUG:teuthology.parallel:result is None 2024-03-29T11:24:26.984 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:24:26.996 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:24:26.996 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:24:26.999 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:24:27.013 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:24:27.024 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:24:27.038 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:24:27.038 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:24:27.053 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:24:27.092 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:24:27.105 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:24:27.105 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:24:27.110 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:24:27.137 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:27.151 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:27.163 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:27.176 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:24:27.178 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:24:27.187 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:24:27.187 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:24:27.195 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:24:27.245 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:24:27.258 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:24:27.258 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:24:27.262 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:24:27.301 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:27.356 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:27.406 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:27.487 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:27.530 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:24:27.531 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:24:27.610 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:27.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:24:27.710 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:27.793 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:24:27.834 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:24:27.834 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:24:27.900 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-03-29T11:24:27.902 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-03-29T11:24:27.948 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:24:27.995 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:24:28.459 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:24:28.468 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:24:28.468 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:24:28.485 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:24:28.496 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-03-29T11:24:28.540 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-03-29T11:24:28.542 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-03-29T11:24:28.600 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-03-29T11:24:28.601 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:24:28.601 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-03-29T11:24:28.626 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-03-29T11:24:28.626 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-03-29T11:24:28.660 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-03-29T11:24:28.660 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi081.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:24:28.660 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:24:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:28.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:24:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:24:28.724 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:24:28.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:24:28.770 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:24:28.812 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:24:28.825 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:24:28.830 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:24:28.950 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:24:28.965 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:24:28.967 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi071.front.sepia.ceph.com,smithi081.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:28:43.918 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi081.front.sepia.ceph.com')] 2024-03-29T11:28:43.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T11:28:43.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:28:44.012 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-29T11:28:44.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-29T11:28:44.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-29T11:28:44.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:28:44.184 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-03-29T11:28:44.223 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-29T11:28:44.224 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:28:44.235 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:28:44.235 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:28:44.236 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:28:44.238 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:28:44.238 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:28:44.266 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:28:44.283 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:28:44.295 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:28:44.312 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-03-29T11:28:44.312 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:28:44.325 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-03-29T11:28:44.342 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-03-29T11:28:44.342 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:28:44.356 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-03-29T11:28:44.358 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:28:44.373 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:28:44.389 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:28:44.418 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:28:44.422 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:28:44.422 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-29T11:28:44.422 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.422 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.423 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.423 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.443 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:28:44.447 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:28:44.447 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-03-29T11:28:44.447 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.447 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.447 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.447 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:28:44.448 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:28:44.457 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:28:44.457 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:28:44.457 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:28:44.457 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:28:44.462 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:28:44.463 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:28:44.463 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:28:44.463 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:28:44.465 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:28:44.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:28:44.609 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:28:44.610 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:28:44.682 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:28:44.683 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:28:44.718 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:28:44.719 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:28:44.719 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:28:44.746 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:28:44.747 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:28:44.790 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:28:44.790 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:28:44.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:28:44.815 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:28:44.820 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:28:44.820 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:28:44.888 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:28:44.895 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:28:44.961 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-03-29T11:28:44.963 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-03-29T11:28:44.967 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:28:45.033 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-03-29T11:28:45.034 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-03-29T11:28:45.199 INFO:teuthology.orchestra.run.smithi071.stdout:74 files removed 2024-03-29T11:28:45.226 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw 2024-03-29T11:28:45.284 INFO:teuthology.orchestra.run.smithi081.stdout:74 files removed 2024-03-29T11:28:45.310 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2024-03-29T11:28:46.066 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 152 kB/s | 82 kB 00:00 2024-03-29T11:28:46.125 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 159 kB/s | 82 kB 00:00 2024-03-29T11:28:46.401 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 35 kB/s | 10 kB 00:00 2024-03-29T11:28:46.550 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T11:28:46.784 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-29T11:28:46.909 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-29T11:28:47.203 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:28:47.328 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:28:48.653 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-29T11:28:48.954 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T11:28:50.053 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T11:28:51.847 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-29T11:28:56.541 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-29T11:28:57.985 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 914 kB/s | 8.0 MB 00:09 2024-03-29T11:28:58.056 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 16 kB 00:00 2024-03-29T11:28:59.150 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-29T11:29:00.244 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T11:29:04.142 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T11:29:04.886 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.1 MB 00:01 2024-03-29T11:29:06.462 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-29T11:29:06.750 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:29:06.758 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:29:06.758 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:29:06.758 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:29:06.758 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:29:06.759 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:29:06.760 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:29:06.761 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:06.762 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:29:06.762 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:29:06.762 INFO:teuthology.orchestra.run.smithi081.stdout:Install 30 Packages 2024-03-29T11:29:06.762 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 2 Packages 2024-03-29T11:29:06.762 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:06.763 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 58 M 2024-03-29T11:29:06.763 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:29:07.760 INFO:teuthology.orchestra.run.smithi081.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.3 MB/s | 5.3 MB 00:00 2024-03-29T11:29:07.826 INFO:teuthology.orchestra.run.smithi081.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 379 kB/s | 25 kB 00:00 2024-03-29T11:29:07.927 INFO:teuthology.orchestra.run.smithi081.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.2 MB/s | 742 kB 00:00 2024-03-29T11:29:08.010 INFO:teuthology.orchestra.run.smithi081.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.8 MB/s | 498 kB 00:00 2024-03-29T11:29:08.119 INFO:teuthology.orchestra.run.smithi081.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 9.7 MB/s | 9.7 MB 00:01 2024-03-29T11:29:08.165 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T11:29:08.211 INFO:teuthology.orchestra.run.smithi081.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-29T11:29:08.336 INFO:teuthology.orchestra.run.smithi081.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-29T11:29:08.361 INFO:teuthology.orchestra.run.smithi081.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 873 kB/s | 131 kB 00:00 2024-03-29T11:29:08.387 INFO:teuthology.orchestra.run.smithi081.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 162 kB 00:00 2024-03-29T11:29:08.412 INFO:teuthology.orchestra.run.smithi081.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.2 MB/s | 320 kB 00:00 2024-03-29T11:29:08.445 INFO:teuthology.orchestra.run.smithi081.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:29:08.471 INFO:teuthology.orchestra.run.smithi081.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:29:08.696 INFO:teuthology.orchestra.run.smithi081.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 423 kB/s | 106 kB 00:00 2024-03-29T11:29:08.754 INFO:teuthology.orchestra.run.smithi081.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 761 kB/s | 44 kB 00:00 2024-03-29T11:29:08.780 INFO:teuthology.orchestra.run.smithi081.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 561 kB/s | 159 kB 00:00 2024-03-29T11:29:08.889 INFO:teuthology.orchestra.run.smithi081.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-29T11:29:08.914 INFO:teuthology.orchestra.run.smithi081.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-29T11:29:08.939 INFO:teuthology.orchestra.run.smithi081.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T11:29:08.964 INFO:teuthology.orchestra.run.smithi081.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:29:08.990 INFO:teuthology.orchestra.run.smithi081.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 694 kB/s | 35 kB 00:00 2024-03-29T11:29:09.015 INFO:teuthology.orchestra.run.smithi081.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 921 kB/s | 46 kB 00:00 2024-03-29T11:29:09.532 INFO:teuthology.orchestra.run.smithi081.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-29T11:29:09.624 INFO:teuthology.orchestra.run.smithi081.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-29T11:29:09.724 INFO:teuthology.orchestra.run.smithi081.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-29T11:29:09.874 INFO:teuthology.orchestra.run.smithi081.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-03-29T11:29:09.900 INFO:teuthology.orchestra.run.smithi081.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 384 kB/s | 67 kB 00:00 2024-03-29T11:29:09.992 INFO:teuthology.orchestra.run.smithi081.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-29T11:29:10.025 INFO:teuthology.orchestra.run.smithi081.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-29T11:29:10.117 INFO:teuthology.orchestra.run.smithi081.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-29T11:29:10.301 INFO:teuthology.orchestra.run.smithi081.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.2 MB/s | 20 MB 00:03 2024-03-29T11:29:10.435 INFO:teuthology.orchestra.run.smithi081.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.2 MB/s | 3.4 MB 00:00 2024-03-29T11:29:10.543 INFO:teuthology.orchestra.run.smithi081.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.4 MB/s | 3.1 MB 00:00 2024-03-29T11:29:10.547 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:10.547 INFO:teuthology.orchestra.run.smithi081.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T11:29:10.548 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:29:10.582 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:29:10.583 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:29:11.059 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:29:11.060 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:29:12.514 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:29:12.598 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:29:12.632 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:29:12.656 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:29:12.680 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:12.697 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:12.864 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:12.918 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:13.171 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:29:13.252 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:29:13.309 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:13.401 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:13.490 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:29:13.527 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:13.591 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:13.630 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:29:13.679 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:29:13.770 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:13.796 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:13.862 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:13.898 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:29:14.028 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:29:14.062 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:29:14.103 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:29:14.136 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:29:14.165 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:29:14.197 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:29:14.507 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:29:14.536 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:14.625 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:14.657 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:29:14.701 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:29:15.167 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:29:15.197 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:15.268 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:15.335 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:29:15.366 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:29:15.793 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:29:15.801 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:29:15.801 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:29:15.801 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:29:15.801 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:29:15.802 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:29:15.803 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:29:15.804 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:29:15.805 INFO:teuthology.orchestra.run.smithi071.stdout:Install 30 Packages 2024-03-29T11:29:15.805 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 2 Packages 2024-03-29T11:29:15.805 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:15.805 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 58 M 2024-03-29T11:29:15.805 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:29:16.397 INFO:teuthology.orchestra.run.smithi071.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 9.8 MB/s | 5.3 MB 00:00 2024-03-29T11:29:16.439 INFO:teuthology.orchestra.run.smithi071.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 605 kB/s | 25 kB 00:00 2024-03-29T11:29:16.531 INFO:teuthology.orchestra.run.smithi071.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-29T11:29:16.581 INFO:teuthology.orchestra.run.smithi071.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 9.8 MB/s | 498 kB 00:00 2024-03-29T11:29:16.931 INFO:teuthology.orchestra.run.smithi071.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-03-29T11:29:16.982 INFO:teuthology.orchestra.run.smithi071.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 903 kB/s | 45 kB 00:00 2024-03-29T11:29:17.048 INFO:teuthology.orchestra.run.smithi071.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T11:29:17.099 INFO:teuthology.orchestra.run.smithi071.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.1 MB/s | 162 kB 00:00 2024-03-29T11:29:17.166 INFO:teuthology.orchestra.run.smithi071.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.7 MB/s | 320 kB 00:00 2024-03-29T11:29:17.233 INFO:teuthology.orchestra.run.smithi071.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-29T11:29:17.291 INFO:teuthology.orchestra.run.smithi071.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:29:17.392 INFO:teuthology.orchestra.run.smithi071.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-29T11:29:17.442 INFO:teuthology.orchestra.run.smithi071.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T11:29:17.467 INFO:teuthology.orchestra.run.smithi071.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-29T11:29:17.507 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:17.534 INFO:teuthology.orchestra.run.smithi071.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-29T11:29:17.562 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:17.584 INFO:teuthology.orchestra.run.smithi071.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-29T11:29:17.618 INFO:teuthology.orchestra.run.smithi071.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:29:17.660 INFO:teuthology.orchestra.run.smithi071.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2024-03-29T11:29:17.693 INFO:teuthology.orchestra.run.smithi071.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:29:17.735 INFO:teuthology.orchestra.run.smithi071.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-29T11:29:17.844 INFO:teuthology.orchestra.run.smithi071.stdout:(21/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 4.9 MB/s | 9.7 MB 00:01 2024-03-29T11:29:18.053 INFO:teuthology.orchestra.run.smithi071.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 9.0 MB/s | 20 MB 00:02 2024-03-29T11:29:18.111 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:18.158 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:18.231 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:18.232 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:29:18.232 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:29:18.232 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:18.267 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:18.278 INFO:teuthology.orchestra.run.smithi071.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2024-03-29T11:29:18.303 INFO:teuthology.orchestra.run.smithi071.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 116 kB/s | 26 kB 00:00 2024-03-29T11:29:18.328 INFO:teuthology.orchestra.run.smithi071.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-03-29T11:29:18.354 INFO:teuthology.orchestra.run.smithi071.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T11:29:18.479 INFO:teuthology.orchestra.run.smithi071.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-29T11:29:18.504 INFO:teuthology.orchestra.run.smithi071.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:29:18.696 INFO:teuthology.orchestra.run.smithi071.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-03-29T11:29:18.730 INFO:teuthology.orchestra.run.smithi071.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-29T11:29:18.788 INFO:teuthology.orchestra.run.smithi071.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-29T11:29:18.989 INFO:teuthology.orchestra.run.smithi071.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T11:29:18.994 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:18.994 INFO:teuthology.orchestra.run.smithi071.stdout:Total 18 MB/s | 58 MB 00:03 2024-03-29T11:29:18.994 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:29:19.030 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:29:19.030 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:29:19.514 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:29:19.515 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:29:21.009 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:29:21.084 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:29:21.118 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:29:21.144 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:29:21.166 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:21.183 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:29:21.325 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:21.387 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:29:21.767 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:29:21.795 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:21.887 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:21.977 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:29:22.013 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:22.059 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:29:22.088 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:29:22.132 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:29:22.201 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:22.224 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:22.271 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:29:22.306 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:29:22.416 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:29:22.457 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:29:22.520 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:29:22.546 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:29:22.575 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:29:22.608 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:29:22.927 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:29:22.955 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:23.044 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:29:23.067 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:29:23.110 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:29:23.563 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:29:23.599 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:23.670 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:29:23.737 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:29:23.768 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:29:25.952 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:25.989 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:26.546 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:29:26.584 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:26.656 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:29:26.657 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:29:26.657 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:29:26.657 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:26.702 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:28.057 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-03-29T11:29:28.058 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:28.969 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:29:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:29:28.994 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:29:28.994 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:29:28.994 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:29:28.994 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:29:28.994 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:28.996 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:29:29.081 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:29:29.081 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:30.249 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:29:30.250 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:29:30.251 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:29:30.252 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:30.253 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:29:30.253 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.357 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:29:31.358 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:29:31.359 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:31.360 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:29:31.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2024-03-29T11:29:32.270 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:29:32.422 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:29:32.423 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:29:32.423 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:29:32.423 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:29:32.423 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-03-29T11:29:32.424 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:32.425 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 48 M 2024-03-29T11:29:32.425 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 204 M 2024-03-29T11:29:32.425 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:29:32.807 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T11:29:32.856 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T11:29:32.891 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:29:33.024 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 163 kB 00:00 2024-03-29T11:29:33.049 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-03-29T11:29:36.364 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-03-29T11:29:36.365 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 47 MB 00:03 2024-03-29T11:29:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-29T11:29:36.459 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:29:36.479 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:29:36.479 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:29:36.869 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:29:36.870 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:29:37.279 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:29:37.307 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:29:37.307 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:29:37.307 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:29:37.307 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:29:37.307 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:29:37.307 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:37.308 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:29:37.398 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:29:37.399 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:37.454 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:29:37.533 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:29:37.572 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:29:37.612 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:29:37.662 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:29:37.686 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:29:38.565 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:29:38.566 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:29:38.567 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:29:38.568 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:29:38.568 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:29:38.568 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:29:38.569 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:29:38.569 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:29:38.569 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:29:38.569 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:29:38.569 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.724 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.725 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:39.726 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:29:40.066 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-test 2024-03-29T11:29:40.630 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:29:40.788 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:29:40.789 INFO:teuthology.orchestra.run.smithi071.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-29T11:29:40.790 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:40.791 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 48 M 2024-03-29T11:29:40.791 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 204 M 2024-03-29T11:29:40.791 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:29:41.103 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:29:41.170 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-29T11:29:41.203 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:29:41.321 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:29:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 367 kB/s | 163 kB 00:00 2024-03-29T11:29:42.178 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:29:42.251 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:29:43.238 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T11:29:43.240 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:43.241 INFO:teuthology.orchestra.run.smithi071.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-29T11:29:43.255 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:29:43.276 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:29:43.277 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:29:43.658 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:29:43.659 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:29:44.238 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:29:44.313 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:29:44.355 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:29:44.395 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:29:44.437 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:29:44.465 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:29:44.789 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:29:44.789 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:29:44.790 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:29:44.790 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:29:44.790 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:29:44.790 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:45.176 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:29:45.535 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2024-03-29T11:29:46.105 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:29:46.258 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:29:46.260 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-29T11:29:46.260 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:29:46.260 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-29T11:29:46.260 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:29:46.261 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:29:46.262 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:29:46.263 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout:Install 36 Packages 2024-03-29T11:29:46.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:46.265 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 30 M 2024-03-29T11:29:46.265 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 107 M 2024-03-29T11:29:46.265 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:29:47.361 INFO:teuthology.orchestra.run.smithi081.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-29T11:29:47.419 INFO:teuthology.orchestra.run.smithi081.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-03-29T11:29:47.470 INFO:teuthology.orchestra.run.smithi081.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.2 MB/s | 2.1 MB 00:00 2024-03-29T11:29:47.536 INFO:teuthology.orchestra.run.smithi081.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T11:29:47.870 INFO:teuthology.orchestra.run.smithi081.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 560 kB/s | 187 kB 00:00 2024-03-29T11:29:48.021 INFO:teuthology.orchestra.run.smithi081.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-03-29T11:29:48.088 INFO:teuthology.orchestra.run.smithi081.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 473 kB/s | 32 kB 00:00 2024-03-29T11:29:48.138 INFO:teuthology.orchestra.run.smithi081.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-29T11:29:48.213 INFO:teuthology.orchestra.run.smithi081.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 515 kB/s | 39 kB 00:00 2024-03-29T11:29:48.322 INFO:teuthology.orchestra.run.smithi081.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-29T11:29:48.564 INFO:teuthology.orchestra.run.smithi081.stdout:(11/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T11:29:48.597 INFO:teuthology.orchestra.run.smithi081.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 4.2 MB/s | 1.1 MB 00:00 2024-03-29T11:29:48.648 INFO:teuthology.orchestra.run.smithi081.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T11:29:48.673 INFO:teuthology.orchestra.run.smithi081.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2024-03-29T11:29:48.715 INFO:teuthology.orchestra.run.smithi081.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-29T11:29:48.937 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:29:48.973 INFO:teuthology.orchestra.run.smithi081.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-29T11:29:49.014 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:29:49.157 INFO:teuthology.orchestra.run.smithi081.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 824 kB/s | 151 kB 00:00 2024-03-29T11:29:49.207 INFO:teuthology.orchestra.run.smithi081.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 886 kB/s | 43 kB 00:00 2024-03-29T11:29:49.299 INFO:teuthology.orchestra.run.smithi081.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-29T11:29:49.375 INFO:teuthology.orchestra.run.smithi081.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-29T11:29:49.417 INFO:teuthology.orchestra.run.smithi081.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-29T11:29:49.458 INFO:teuthology.orchestra.run.smithi081.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-03-29T11:29:49.500 INFO:teuthology.orchestra.run.smithi081.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 557 kB/s | 23 kB 00:00 2024-03-29T11:29:49.576 INFO:teuthology.orchestra.run.smithi081.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-29T11:29:49.626 INFO:teuthology.orchestra.run.smithi081.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 394 kB/s | 20 kB 00:00 2024-03-29T11:29:49.676 INFO:teuthology.orchestra.run.smithi081.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 930 kB/s | 46 kB 00:00 2024-03-29T11:29:49.701 INFO:teuthology.orchestra.run.smithi081.stdout:(27/36): python3-requests-2.25.1-8.el9.noarch.r 121 kB/s | 125 kB 00:01 2024-03-29T11:29:49.727 INFO:teuthology.orchestra.run.smithi081.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:29:49.752 INFO:teuthology.orchestra.run.smithi081.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.4 MB/s | 272 kB 00:00 2024-03-29T11:29:49.777 INFO:teuthology.orchestra.run.smithi081.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 331 kB/s | 16 kB 00:00 2024-03-29T11:29:49.802 INFO:teuthology.orchestra.run.smithi081.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-03-29T11:29:49.828 INFO:teuthology.orchestra.run.smithi081.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-03-29T11:29:49.853 INFO:teuthology.orchestra.run.smithi081.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T11:29:49.904 INFO:teuthology.orchestra.run.smithi081.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-29T11:29:49.929 INFO:teuthology.orchestra.run.smithi081.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 268 kB/s | 20 kB 00:00 2024-03-29T11:29:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:29:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:29:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:29:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:29:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:29:51.394 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:29:51.758 INFO:teuthology.orchestra.run.smithi071.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:29:51.759 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:29:51.759 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:29:51.759 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:51.759 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:29:52.039 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph 2024-03-29T11:29:52.623 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:39 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:29:52.798 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:29:52.801 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-29T11:29:52.801 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:29:52.801 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:29:52.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:29:52.803 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:29:52.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:29:52.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:29:52.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:29:52.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:29:52.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:29:52.804 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:29:52.805 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:29:52.805 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:29:52.805 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:29:52.805 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:29:52.805 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:29:52.805 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:29:52.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:29:52.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:29:52.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:29:52.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:29:52.806 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:29:52.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:29:52.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:29:52.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:29:52.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:29:52.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:29:52.807 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:29:52.808 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:52.808 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:29:52.808 INFO:teuthology.orchestra.run.smithi071.stdout:========================================================================================= 2024-03-29T11:29:52.808 INFO:teuthology.orchestra.run.smithi071.stdout:Install 36 Packages 2024-03-29T11:29:52.808 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:29:52.809 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 30 M 2024-03-29T11:29:52.810 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 107 M 2024-03-29T11:29:52.810 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:29:53.398 INFO:teuthology.orchestra.run.smithi071.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 22 kB/s | 6.6 kB 00:00 2024-03-29T11:29:54.054 INFO:teuthology.orchestra.run.smithi081.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 717 kB/s | 4.7 MB 00:06 2024-03-29T11:29:54.059 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:29:54.059 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.8 MB/s | 30 MB 00:07 2024-03-29T11:29:54.084 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:29:54.126 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:29:54.126 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:29:54.534 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:29:54.534 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:29:54.815 INFO:teuthology.orchestra.run.smithi071.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 891 kB/s | 1.5 MB 00:01 2024-03-29T11:29:55.149 INFO:teuthology.orchestra.run.smithi071.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.0 MB/s | 2.1 MB 00:02 2024-03-29T11:29:55.382 INFO:teuthology.orchestra.run.smithi071.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.0 MB/s | 247 kB 00:00 2024-03-29T11:29:55.643 INFO:teuthology.orchestra.run.smithi071.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 717 kB/s | 187 kB 00:00 2024-03-29T11:29:55.718 INFO:teuthology.orchestra.run.smithi071.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-29T11:29:55.777 INFO:teuthology.orchestra.run.smithi071.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 541 kB/s | 32 kB 00:00 2024-03-29T11:29:55.835 INFO:teuthology.orchestra.run.smithi071.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-03-29T11:29:55.859 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:29:55.894 INFO:teuthology.orchestra.run.smithi071.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 663 kB/s | 39 kB 00:00 2024-03-29T11:29:55.923 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:29:55.972 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:29:55.986 INFO:teuthology.orchestra.run.smithi071.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2024-03-29T11:29:56.000 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:29:56.070 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:29:56.086 INFO:teuthology.orchestra.run.smithi071.stdout:(11/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 4.7 MB 00:02 2024-03-29T11:29:56.116 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:29:56.120 INFO:teuthology.orchestra.run.smithi071.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-03-29T11:29:56.145 INFO:teuthology.orchestra.run.smithi071.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-29T11:29:56.148 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:29:56.170 INFO:teuthology.orchestra.run.smithi071.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:29:56.195 INFO:teuthology.orchestra.run.smithi071.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:29:56.240 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:29:56.287 INFO:teuthology.orchestra.run.smithi071.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-29T11:29:56.313 INFO:teuthology.orchestra.run.smithi071.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 881 kB/s | 125 kB 00:00 2024-03-29T11:29:56.326 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:29:56.368 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:29:56.439 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:29:56.493 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:29:56.541 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:29:56.597 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:29:56.621 INFO:teuthology.orchestra.run.smithi071.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-03-29T11:29:56.652 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:29:56.746 INFO:teuthology.orchestra.run.smithi071.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 329 kB/s | 151 kB 00:00 2024-03-29T11:29:56.860 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:29:56.897 INFO:teuthology.orchestra.run.smithi071.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 624 kB/s | 172 kB 00:00 2024-03-29T11:29:56.911 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:29:56.942 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:29:56.977 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:29:56.989 INFO:teuthology.orchestra.run.smithi071.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-29T11:29:57.006 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:29:57.014 INFO:teuthology.orchestra.run.smithi071.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 91 kB/s | 11 kB 00:00 2024-03-29T11:29:57.081 INFO:teuthology.orchestra.run.smithi071.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-29T11:29:57.082 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:29:57.117 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:29:57.139 INFO:teuthology.orchestra.run.smithi071.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2024-03-29T11:29:57.141 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:29:57.173 INFO:teuthology.orchestra.run.smithi071.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-29T11:29:57.174 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:29:57.209 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:29:57.240 INFO:teuthology.orchestra.run.smithi071.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-29T11:29:57.243 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:29:57.274 INFO:teuthology.orchestra.run.smithi071.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-29T11:29:57.333 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:29:57.349 INFO:teuthology.orchestra.run.smithi071.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-03-29T11:29:57.374 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:29:57.407 INFO:teuthology.orchestra.run.smithi071.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-29T11:29:57.445 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:29:57.449 INFO:teuthology.orchestra.run.smithi071.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T11:29:57.516 INFO:teuthology.orchestra.run.smithi071.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 828 kB/s | 90 kB 00:00 2024-03-29T11:29:57.550 INFO:teuthology.orchestra.run.smithi071.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-29T11:29:57.581 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:29:57.650 INFO:teuthology.orchestra.run.smithi071.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-29T11:29:57.714 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:29:57.754 INFO:teuthology.orchestra.run.smithi071.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-29T11:29:57.784 INFO:teuthology.orchestra.run.smithi071.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-03-29T11:29:57.873 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:29:57.904 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:29:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:29:57.929 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:29:57.929 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:29:57.929 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:29:57.929 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:29:57.929 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:59.347 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:29:59.374 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:29:59.374 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:29:59.374 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:29:59.374 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:29:59.374 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:29:59.374 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:29:59.814 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:29:59.837 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:29:59.838 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:29:59.838 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:29:59.838 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:29:59.838 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:29:59.838 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:00.040 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:00.074 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:00.074 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:00.074 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:30:00.074 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:00.074 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:00.074 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:00.130 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:30:01.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:30:01.959 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:30:01.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:30:01.961 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:30:01.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:30:02.050 INFO:teuthology.orchestra.run.smithi071.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.3 MB/s | 17 MB 00:07 2024-03-29T11:30:02.054 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:02.054 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.2 MB/s | 30 MB 00:09 2024-03-29T11:30:02.079 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:02.121 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:02.122 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:02.500 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:02.500 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:02.980 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:30:02.980 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:02.981 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:30:02.982 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:30:02.983 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:30:02.984 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:02.985 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:03.315 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base 2024-03-29T11:30:03.861 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:03.879 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:03.936 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:30:03.979 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:03.987 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:30:04.022 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:30:04.031 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:04.032 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:04.032 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:04.091 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:30:04.099 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install cephadm 2024-03-29T11:30:04.128 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:30:04.162 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:30:04.252 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:30:04.331 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:30:04.372 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:30:04.443 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:30:04.498 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:30:04.539 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:30:04.585 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:30:04.640 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:30:04.657 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:04.809 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:04.810 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 747 k 2024-03-29T11:30:04.811 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 754 k 2024-03-29T11:30:04.811 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:04.823 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:30:04.882 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:30:04.912 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:30:04.947 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:30:04.977 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:30:05.053 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:30:05.087 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:05.121 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:30:05.160 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:30:05.165 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.1 MB/s | 747 kB 00:00 2024-03-29T11:30:05.166 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:05.166 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.0 MB/s | 747 kB 00:00 2024-03-29T11:30:05.167 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:05.172 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:05.172 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:05.177 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:05.178 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:05.196 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:30:05.239 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:30:05.249 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:05.327 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:30:05.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:30:05.433 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:30:05.569 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:30:05.686 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:05.714 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:05.787 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:05.843 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:30:05.866 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:30:05.891 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:30:05.892 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:05.892 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:30:05.892 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:30:05.892 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:30:05.892 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:06.162 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:06.442 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:06.442 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:06.442 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:06.442 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:06.442 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:06.653 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:30:07.224 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:07.317 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:30:07.343 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:30:07.344 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:07.344 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:30:07.344 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:30:07.344 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:30:07.344 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:07.379 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:07.379 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 145 k 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 436 k 2024-03-29T11:30:07.380 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:07.714 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 436 kB/s | 145 kB 00:00 2024-03-29T11:30:07.715 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:07.715 INFO:teuthology.orchestra.run.smithi081.stdout:Total 434 kB/s | 145 kB 00:00 2024-03-29T11:30:07.715 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:07.723 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:07.723 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:07.776 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:07.776 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:07.784 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:30:07.807 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:30:07.807 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:07.807 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:30:07.807 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:30:07.808 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:30:07.808 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:07.915 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:07.973 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:07.998 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:07.999 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:07.999 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:30:07.999 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:08.018 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:08.041 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:30:08.041 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:08.041 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:30:08.042 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:08.042 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:30:08.042 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:08.107 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:08.986 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:08.986 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:08.987 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:08.987 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:09.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:30:09.767 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:30:09.768 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:30:09.769 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:30:09.769 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:30:09.769 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:30:09.769 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:30:09.769 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:30:09.770 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:30:09.818 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:09.969 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:09.970 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:09.970 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:10.044 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:30:10.605 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:10.677 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:30:10.677 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:10.677 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:10.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:10.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:10.677 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:30:10.678 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:30:10.679 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:10.680 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:10.756 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:10.757 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-29T11:30:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:10.757 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-29T11:30:10.757 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:10.757 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:30:10.758 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:10.759 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:10.759 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-29T11:30:10.759 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-03-29T11:30:10.759 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:10.759 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 6.9 M 2024-03-29T11:30:10.759 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-03-29T11:30:10.760 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:11.037 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-base 2024-03-29T11:30:11.385 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 160 kB/s | 31 kB 00:00 2024-03-29T11:30:11.410 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 69 kB/s | 15 kB 00:00 2024-03-29T11:30:11.552 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-29T11:30:11.601 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:11.602 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T11:30:11.647 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.9 MB/s | 1.8 MB 00:00 2024-03-29T11:30:11.698 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:11.751 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:11.752 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:30:11.752 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:11.811 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install cephadm 2024-03-29T11:30:11.819 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 128 kB/s | 19 kB 00:00 2024-03-29T11:30:12.353 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-29T11:30:12.369 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:12.495 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 614 kB/s | 548 kB 00:00 2024-03-29T11:30:12.519 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:12.519 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:12.519 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:12.519 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:12.519 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:12.519 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:30:12.520 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:12.520 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:12.520 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:12.520 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:30:12.520 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:12.520 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 868 kB/s | 144 kB 00:00 2024-03-29T11:30:12.521 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 747 k 2024-03-29T11:30:12.521 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 754 k 2024-03-29T11:30:12.521 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:12.579 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 542 kB/s | 31 kB 00:00 2024-03-29T11:30:12.604 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.3 MB/s | 188 kB 00:00 2024-03-29T11:30:12.646 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-03-29T11:30:12.651 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:12.651 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.6 MB/s | 6.9 MB 00:01 2024-03-29T11:30:12.669 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:12.686 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:12.879 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:12.880 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:12.914 INFO:teuthology.orchestra.run.smithi071.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.9 MB/s | 747 kB 00:00 2024-03-29T11:30:12.915 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:12.915 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.8 MB/s | 747 kB 00:00 2024-03-29T11:30:12.915 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:12.922 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:12.922 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:12.925 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:12.926 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:13.001 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:13.437 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:13.437 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:13.526 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:13.529 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:30:13.573 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:30:13.619 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:30:13.650 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:30:13.751 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:30:13.873 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:30:13.886 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:13.960 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:30:14.004 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:30:14.033 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:14.065 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:30:14.173 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:30:14.173 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:14.173 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:14.173 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:14.173 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:14.173 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:14.306 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:30:14.340 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:14.401 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:30:14.970 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:15.126 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:15.127 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:15.128 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:15.128 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:30:15.128 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:15.128 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 145 k 2024-03-29T11:30:15.128 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 436 k 2024-03-29T11:30:15.128 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:30:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:30:15.477 INFO:teuthology.orchestra.run.smithi071.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 416 kB/s | 145 kB 00:00 2024-03-29T11:30:15.478 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:15.478 INFO:teuthology.orchestra.run.smithi071.stdout:Total 415 kB/s | 145 kB 00:00 2024-03-29T11:30:15.478 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:15.485 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:15.486 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:15.537 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:15.538 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:30:15.543 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:30:15.544 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:30:15.544 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:30:15.544 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:30:15.544 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:15.544 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:15.698 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:15.762 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:15.786 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:15.786 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:15.786 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:30:15.787 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:15.827 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:30:16.413 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:16.570 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-29T11:30:16.571 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-29T11:30:16.571 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:16.571 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:30:16.572 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:16.573 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:16.573 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-29T11:30:16.573 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-03-29T11:30:16.573 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:16.573 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 37 M 2024-03-29T11:30:16.573 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 209 M 2024-03-29T11:30:16.574 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:16.749 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:30:16.749 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:16.749 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:16.750 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:16.750 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:16.750 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:16.996 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr 2024-03-29T11:30:17.095 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 146 kB/s | 32 kB 00:00 2024-03-29T11:30:17.153 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 298 kB/s | 17 kB 00:00 2024-03-29T11:30:17.254 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 341 kB/s | 34 kB 00:00 2024-03-29T11:30:17.574 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:17.604 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-03-29T11:30:17.673 INFO:teuthology.orchestra.run.smithi071.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:17.725 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:17.726 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:30:17.726 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:17.771 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.2 MB/s | 4.7 MB 00:00 2024-03-29T11:30:17.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:30:17.988 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.0 MB/s | 217 kB 00:00 2024-03-29T11:30:18.055 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-29T11:30:18.347 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:18.498 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:18.499 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-29T11:30:18.499 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:18.499 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-29T11:30:18.499 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:30:18.500 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout:====================================================================================== 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.9 M 2024-03-29T11:30:18.501 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-29T11:30:18.502 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:18.605 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-29T11:30:18.705 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-29T11:30:18.866 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 755 kB/s | 195 kB 00:00 2024-03-29T11:30:18.889 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-29T11:30:19.147 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 85 kB/s | 15 kB 00:00 2024-03-29T11:30:19.239 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 115 kB/s | 31 kB 00:00 2024-03-29T11:30:19.306 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:30:19.331 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T11:30:19.373 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.4 MB/s | 1.8 MB 00:00 2024-03-29T11:30:19.532 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-29T11:30:19.665 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-03-29T11:30:19.749 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T11:30:19.824 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 410 kB/s | 31 kB 00:00 2024-03-29T11:30:19.874 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.0 MB/s | 2.0 MB 00:00 2024-03-29T11:30:19.925 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T11:30:19.983 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 455 kB/s | 144 kB 00:00 2024-03-29T11:30:19.986 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:19.986 INFO:teuthology.orchestra.run.smithi071.stdout:Total 4.6 MB/s | 6.9 MB 00:01 2024-03-29T11:30:20.003 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:20.020 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:20.020 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:20.194 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:20.195 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:20.728 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:20.800 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:30:20.835 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:30:20.881 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:30:20.912 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:30:21.022 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:30:21.144 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:30:21.247 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:30:21.291 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:30:21.329 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:21.352 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:30:21.373 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.6 MB/s | 7.4 MB 00:04 2024-03-29T11:30:21.375 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:21.375 INFO:teuthology.orchestra.run.smithi081.stdout:Total 7.7 MB/s | 37 MB 00:04 2024-03-29T11:30:21.418 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:21.444 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:21.444 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:21.591 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:30:21.627 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:21.811 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:21.811 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:22.386 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:30:22.387 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:30:22.516 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:22.579 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:30:22.610 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:30:22.709 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:22.728 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:30:22.728 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:22.728 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:22.728 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:30:22.728 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:22.728 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:22.729 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:22.755 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:30:22.931 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:22.980 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:22.992 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:30:23.061 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:30:23.564 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:23.714 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:30:23.726 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:30:23.727 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:23.728 INFO:teuthology.orchestra.run.smithi071.stdout:============================================================================================ 2024-03-29T11:30:23.729 INFO:teuthology.orchestra.run.smithi071.stdout:Install 12 Packages 2024-03-29T11:30:23.729 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:23.729 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 37 M 2024-03-29T11:30:23.729 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 209 M 2024-03-29T11:30:23.729 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:23.817 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:30:24.065 INFO:teuthology.orchestra.run.smithi071.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-03-29T11:30:24.098 INFO:teuthology.orchestra.run.smithi071.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 541 kB/s | 17 kB 00:00 2024-03-29T11:30:24.140 INFO:teuthology.orchestra.run.smithi071.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 848 kB/s | 34 kB 00:00 2024-03-29T11:30:24.207 INFO:teuthology.orchestra.run.smithi071.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:30:24.374 INFO:teuthology.orchestra.run.smithi071.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:30:24.441 INFO:teuthology.orchestra.run.smithi071.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-29T11:30:24.474 INFO:teuthology.orchestra.run.smithi071.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T11:30:24.817 INFO:teuthology.orchestra.run.smithi071.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-29T11:30:24.967 INFO:teuthology.orchestra.run.smithi071.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-29T11:30:24.992 INFO:teuthology.orchestra.run.smithi071.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:30:25.074 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:30:25.109 INFO:teuthology.orchestra.run.smithi071.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:30:25.626 INFO:teuthology.orchestra.run.smithi071.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.3 MB/s | 7.4 MB 00:01 2024-03-29T11:30:25.628 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:25.628 INFO:teuthology.orchestra.run.smithi071.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-29T11:30:25.670 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:25.694 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:25.694 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:26.058 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:26.059 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:26.088 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:30:26.129 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:26.765 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:26.840 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:30:26.880 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:30:26.973 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:27.016 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:30:27.193 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:27.232 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:27.308 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:30:28.077 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:30:28.836 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:30:29.308 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:30:29.370 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:30:29.370 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:29.370 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:29.370 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:29.370 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:30:29.370 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:29.371 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:29.781 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:30:30.351 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:30.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:30:30.389 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:30.503 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:30:30.505 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-29T11:30:30.506 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2024-03-29T11:30:30.507 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:30.507 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.0 M 2024-03-29T11:30:30.507 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-03-29T11:30:30.507 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:30.971 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 18 kB 00:00 2024-03-29T11:30:30.996 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 24 kB 00:00 2024-03-29T11:30:31.046 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:30:31.071 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:30:31.097 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:30:31.122 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 849 kB/s | 43 kB 00:00 2024-03-29T11:30:31.222 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 144 kB/s | 52 kB 00:00 2024-03-29T11:30:31.264 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T11:30:31.290 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 90 kB/s | 15 kB 00:00 2024-03-29T11:30:31.332 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.8 MB/s | 203 kB 00:00 2024-03-29T11:30:31.407 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-03-29T11:30:31.424 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-03-29T11:30:31.449 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 766 kB/s | 90 kB 00:00 2024-03-29T11:30:31.453 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:31.453 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:30:31.453 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:31.467 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:31.467 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:31.559 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:31.559 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:32.083 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:32.255 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:30:32.308 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:30:32.365 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:30:32.401 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:30:32.443 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:30:32.544 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:30:32.645 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:30:32.694 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:30:32.940 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:30:32.941 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:30:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:30:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:33.415 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:30:33.415 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:33.416 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:33.419 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:30:33.466 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:30:33.491 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:30:33.531 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:30:33.553 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:30:33.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:30:34.357 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:30:34.358 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:30:34.369 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:34.523 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:34.524 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T11:30:34.524 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:34.524 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:30:34.525 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:30:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:34.526 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:34.526 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T11:30:34.526 INFO:teuthology.orchestra.run.smithi071.stdout:Install 13 Packages 2024-03-29T11:30:34.526 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:34.526 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 2.0 M 2024-03-29T11:30:34.527 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 26 M 2024-03-29T11:30:34.527 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:34.715 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:30:34.716 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:30:34.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:30:34.717 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:30:34.717 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:34.788 INFO:teuthology.orchestra.run.smithi071.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 161 kB/s | 24 kB 00:00 2024-03-29T11:30:34.813 INFO:teuthology.orchestra.run.smithi071.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 102 kB/s | 18 kB 00:00 2024-03-29T11:30:34.864 INFO:teuthology.orchestra.run.smithi071.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:30:34.889 INFO:teuthology.orchestra.run.smithi071.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:30:34.914 INFO:teuthology.orchestra.run.smithi071.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:30:34.939 INFO:teuthology.orchestra.run.smithi071.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T11:30:34.965 INFO:teuthology.orchestra.run.smithi071.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 159 kB/s | 52 kB 00:00 2024-03-29T11:30:34.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:30:35.281 INFO:teuthology.orchestra.run.smithi071.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2024-03-29T11:30:35.332 INFO:teuthology.orchestra.run.smithi071.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 77 kB/s | 32 kB 00:00 2024-03-29T11:30:35.507 INFO:teuthology.orchestra.run.smithi071.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 338 kB/s | 59 kB 00:00 2024-03-29T11:30:35.546 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:35.549 INFO:teuthology.orchestra.run.smithi071.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 347 kB/s | 203 kB 00:00 2024-03-29T11:30:35.649 INFO:teuthology.orchestra.run.smithi071.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 631 kB/s | 90 kB 00:00 2024-03-29T11:30:35.699 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:35.699 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-29T11:30:35.699 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-29T11:30:35.700 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-03-29T11:30:35.701 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:35.701 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 6.8 M 2024-03-29T11:30:35.701 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 33 M 2024-03-29T11:30:35.701 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:35.791 INFO:teuthology.orchestra.run.smithi071.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-29T11:30:35.795 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:35.795 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T11:30:35.796 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:35.809 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:35.809 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:35.900 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:35.900 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:36.309 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-29T11:30:36.404 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:36.410 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-29T11:30:36.460 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 410 kB/s | 138 kB 00:00 2024-03-29T11:30:36.576 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:30:36.641 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:30:36.698 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:30:36.726 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:30:36.767 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:30:36.861 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:30:36.910 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-03-29T11:30:36.935 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 107 kB/s | 51 kB 00:00 2024-03-29T11:30:36.960 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:30:37.018 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:30:37.119 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 679 kB/s | 548 kB 00:00 2024-03-29T11:30:37.122 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:37.123 INFO:teuthology.orchestra.run.smithi081.stdout:Total 4.8 MB/s | 6.8 MB 00:01 2024-03-29T11:30:37.137 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:37.148 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:37.148 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:37.238 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:37.239 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:37.546 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:37.715 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:30:37.721 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:30:37.762 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:30:37.766 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:30:37.799 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:30:37.847 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:30:37.869 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:30:38.277 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:30:38.366 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:30:38.428 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:30:38.451 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:30:38.654 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:30:38.654 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:30:38.654 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:30:38.654 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:30:38.654 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:30:38.654 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:30:38.655 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:30:39.011 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:30:39.011 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:39.011 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:39.011 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:30:39.012 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:30:39.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:30:39.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:30:39.013 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:30:39.013 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:39.013 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:39.248 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:30:39.248 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:30:39.248 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:30:39.248 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:30:39.248 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:30:39.248 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:30:39.281 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:30:39.556 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:30:39.556 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:39.556 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:39.557 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:39.557 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:30:39.557 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:30:39.557 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:30:39.557 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:30:39.557 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:30:39.558 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:39.558 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:39.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2024-03-29T11:30:39.846 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:39.998 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:30:39.999 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================== 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout:Install 6 Packages 2024-03-29T11:30:40.000 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:40.001 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 6.8 M 2024-03-29T11:30:40.001 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 33 M 2024-03-29T11:30:40.001 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:40.370 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:40.434 INFO:teuthology.orchestra.run.smithi071.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-29T11:30:40.521 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:40.522 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-29T11:30:40.523 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:40.523 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 964 k 2024-03-29T11:30:40.523 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.8 M 2024-03-29T11:30:40.523 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:40.784 INFO:teuthology.orchestra.run.smithi071.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 255 kB/s | 138 kB 00:00 2024-03-29T11:30:40.814 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 270 kB/s | 79 kB 00:00 2024-03-29T11:30:41.189 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 885 kB 00:00 2024-03-29T11:30:41.190 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:41.191 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.4 MB/s | 964 kB 00:00 2024-03-29T11:30:41.191 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:41.211 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:41.211 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:41.286 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:41.286 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:41.480 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:41.526 INFO:teuthology.orchestra.run.smithi071.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 5.8 MB 00:01 2024-03-29T11:30:41.615 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:30:41.681 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:30:41.743 INFO:teuthology.orchestra.run.smithi071.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 419 kB/s | 548 kB 00:01 2024-03-29T11:30:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:30:42.380 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:30:42.681 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:30:42.681 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:42.681 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:42.681 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:30:42.682 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:42.682 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:42.684 INFO:teuthology.orchestra.run.smithi071.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 44 kB/s | 51 kB 00:01 2024-03-29T11:30:42.969 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-volume 2024-03-29T11:30:43.543 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:43.701 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:43.703 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:43.703 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-29T11:30:43.703 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:43.703 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:43.704 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:30:43.704 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-29T11:30:43.704 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:30:43.704 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:43.704 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:43.704 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:43.705 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-29T11:30:43.705 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:43.705 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 332 k 2024-03-29T11:30:43.705 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.5 M 2024-03-29T11:30:43.705 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:43.844 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 489 kB/s | 69 kB 00:00 2024-03-29T11:30:44.351 INFO:teuthology.orchestra.run.smithi071.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 16 kB/s | 58 kB 00:03 2024-03-29T11:30:44.353 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:44.353 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.6 MB/s | 6.8 MB 00:04 2024-03-29T11:30:44.369 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:44.380 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:44.380 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:44.471 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:44.472 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:44.561 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 307 kB/s | 264 kB 00:00 2024-03-29T11:30:44.563 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:44.563 INFO:teuthology.orchestra.run.smithi081.stdout:Total 386 kB/s | 332 kB 00:00 2024-03-29T11:30:44.563 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:44.575 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:44.575 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:44.614 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:44.615 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:44.765 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:44.770 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:44.945 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:30:44.951 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:30:44.981 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:30:44.997 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:30:44.997 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:44.997 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:30:44.998 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:45.003 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:30:45.536 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:30:45.631 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:30:45.700 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:30:45.731 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:30:45.737 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:46.092 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:46.455 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel 2024-03-29T11:30:46.492 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:30:46.493 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:30:46.493 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:30:46.493 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:30:46.493 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:30:46.493 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:30:46.801 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:30:46.801 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:46.802 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:47.032 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:47.067 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-fuse 2024-03-29T11:30:47.181 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:30:47.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:47.183 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 127 k 2024-03-29T11:30:47.183 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 456 k 2024-03-29T11:30:47.183 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:47.622 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:47.693 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 248 kB/s | 127 kB 00:00 2024-03-29T11:30:47.694 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:47.694 INFO:teuthology.orchestra.run.smithi081.stdout:Total 247 kB/s | 127 kB 00:00 2024-03-29T11:30:47.694 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:47.699 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:47.700 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:47.755 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:47.756 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:47.770 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:47.771 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:47.771 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:47.771 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:47.771 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 964 k 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 2.8 M 2024-03-29T11:30:47.772 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:47.900 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:47.910 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 570 kB/s | 79 kB 00:00 2024-03-29T11:30:48.017 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:48.521 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:48.527 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-03-29T11:30:48.528 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:48.528 INFO:teuthology.orchestra.run.smithi071.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-03-29T11:30:48.529 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:48.550 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:48.550 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:48.625 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:48.625 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:48.800 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:48.829 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:48.830 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:48.830 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:48.830 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:48.830 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:48.830 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:48.944 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:30:49.018 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:30:49.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2024-03-29T11:30:49.630 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:30:49.630 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:30:49.685 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:49.786 INFO:teuthology.orchestra.run.smithi081.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:49.842 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:49.843 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:49.843 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:49.914 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2024-03-29T11:30:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:30:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:49.924 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:30:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:49.924 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:50.185 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-volume 2024-03-29T11:30:50.478 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:50.630 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:50.631 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:50.632 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:50.632 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:30:50.632 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:50.632 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 31 k 2024-03-29T11:30:50.632 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 140 k 2024-03-29T11:30:50.632 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:50.759 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:50.916 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout:Install 2 Packages 2024-03-29T11:30:50.917 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:50.918 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 332 k 2024-03-29T11:30:50.918 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 1.5 M 2024-03-29T11:30:50.918 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:51.038 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 77 kB/s | 31 kB 00:00 2024-03-29T11:30:51.039 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:51.039 INFO:teuthology.orchestra.run.smithi081.stdout:Total 76 kB/s | 31 kB 00:00 2024-03-29T11:30:51.039 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:51.040 INFO:teuthology.orchestra.run.smithi071.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 567 kB/s | 69 kB 00:00 2024-03-29T11:30:51.042 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:51.042 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:51.075 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:51.075 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:51.202 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:51.310 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:51.607 INFO:teuthology.orchestra.run.smithi071.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 383 kB/s | 264 kB 00:00 2024-03-29T11:30:51.608 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:51.608 INFO:teuthology.orchestra.run.smithi071.stdout:Total 481 kB/s | 332 kB 00:00 2024-03-29T11:30:51.608 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:51.617 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:51.617 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:51.661 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:51.661 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:51.693 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:51.813 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:51.979 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:51.979 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:51.979 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:51.979 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:51.979 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:51.979 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:51.998 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:30:52.042 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:30:52.059 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:30:52.060 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:30:52.061 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:30:52.061 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:52.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2024-03-29T11:30:52.737 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:52.793 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:30:52.834 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:52.885 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:52.886 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:52.886 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:52.934 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2024-03-29T11:30:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:30:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:53.125 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:30:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:30:53.125 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:53.126 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:53.454 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados-devel 2024-03-29T11:30:53.497 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:53.594 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:53.645 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:53.646 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:53.646 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:53.696 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados 2024-03-29T11:30:54.033 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:54.183 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:54.184 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:54.184 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T11:30:54.184 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 127 k 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 456 k 2024-03-29T11:30:54.185 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:54.258 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:54.356 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:54.407 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:54.408 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:54.408 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:54.469 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw 2024-03-29T11:30:54.494 INFO:teuthology.orchestra.run.smithi071.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 410 kB/s | 127 kB 00:00 2024-03-29T11:30:54.495 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:54.495 INFO:teuthology.orchestra.run.smithi071.stdout:Total 408 kB/s | 127 kB 00:00 2024-03-29T11:30:54.495 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:54.501 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:54.501 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:54.557 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:54.557 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:54.708 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:54.819 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:55.038 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:55.137 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:55.187 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:55.187 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:55.188 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:55.241 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:55.243 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs 2024-03-29T11:30:55.550 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:55.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:55.550 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:55.550 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:55.550 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:55.550 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:55.791 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs2 2024-03-29T11:30:55.809 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:55.906 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:55.958 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:55.959 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:56.010 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd 2024-03-29T11:30:56.369 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:56.470 INFO:teuthology.orchestra.run.smithi071.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:56.526 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:56.527 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:30:56.527 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:56.572 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:56.586 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install libcephfs-devel 2024-03-29T11:30:56.670 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:56.721 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:56.722 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-29T11:30:56.722 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:56.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2024-03-29T11:30:57.151 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:57.304 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:57.305 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 31 k 2024-03-29T11:30:57.306 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 140 k 2024-03-29T11:30:57.306 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:30:57.329 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:57.478 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:57.479 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:57.479 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-29T11:30:57.479 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:57.479 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:57.479 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 85 k 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 231 k 2024-03-29T11:30:57.480 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:57.567 INFO:teuthology.orchestra.run.smithi071.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 119 kB/s | 31 kB 00:00 2024-03-29T11:30:57.567 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:57.567 INFO:teuthology.orchestra.run.smithi071.stdout:Total 119 kB/s | 31 kB 00:00 2024-03-29T11:30:57.567 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:30:57.575 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:30:57.575 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:30:57.595 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:30:57.595 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:30:57.719 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:30:57.840 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:57.925 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 192 kB/s | 85 kB 00:00 2024-03-29T11:30:57.925 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:30:57.926 INFO:teuthology.orchestra.run.smithi081.stdout:Total 191 kB/s | 85 kB 00:00 2024-03-29T11:30:57.926 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:30:57.933 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:30:57.933 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:30:57.989 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:30:57.989 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:30:58.140 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:30:58.232 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:58.248 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:58.497 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:58.497 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:58.497 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:30:58.498 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:58.498 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:30:58.498 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:58.651 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:58.664 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librados2 2024-03-29T11:30:58.964 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:30:58.964 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:58.964 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:30:58.964 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:30:58.964 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:58.964 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:30:59.227 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:30:59.237 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror 2024-03-29T11:30:59.324 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:30:59.375 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:30:59.376 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:30:59.377 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:30:59.425 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install librbd1 2024-03-29T11:30:59.814 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:55 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:30:59.968 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:30:59.969 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:59.969 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-29T11:30:59.969 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:59.969 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.1 M 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 13 M 2024-03-29T11:30:59.970 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:30:59.984 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:00.082 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:00.133 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:00.134 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:31:00.134 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:00.186 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rados 2024-03-29T11:31:00.378 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.6 MB/s | 3.1 MB 00:00 2024-03-29T11:31:00.378 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:00.379 INFO:teuthology.orchestra.run.smithi081.stdout:Total 7.6 MB/s | 3.1 MB 00:00 2024-03-29T11:31:00.379 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:31:00.387 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:31:00.388 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:31:00.462 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:31:00.462 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:31:00.749 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:00.847 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:00.899 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:00.899 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:31:00.900 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:31:00.900 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:00.945 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rgw 2024-03-29T11:31:00.962 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:00.986 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:00.986 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:31:00.986 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:00.986 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:00.986 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:01.507 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:01.606 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:01.659 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:01.660 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:31:01.660 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:01.704 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-cephfs 2024-03-29T11:31:02.137 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:02.138 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:02.138 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:31:02.138 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:02.138 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:02.138 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:31:02.271 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:02.370 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:02.389 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd 2024-03-29T11:31:02.422 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:02.423 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:31:02.423 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:02.469 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install python3-rbd 2024-03-29T11:31:02.966 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:29:04 AM UTC. 2024-03-29T11:31:03.035 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:03.115 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-29T11:31:03.115 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:31:03.115 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 171 k 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 494 k 2024-03-29T11:31:03.116 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-29T11:31:03.132 INFO:teuthology.orchestra.run.smithi071.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:31:03.184 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:03.185 INFO:teuthology.orchestra.run.smithi071.stdout:Nothing to do. 2024-03-29T11:31:03.186 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:03.232 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-fuse 2024-03-29T11:31:03.348 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 741 kB/s | 171 kB 00:00 2024-03-29T11:31:03.349 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:03.349 INFO:teuthology.orchestra.run.smithi081.stdout:Total 735 kB/s | 171 kB 00:00 2024-03-29T11:31:03.349 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-29T11:31:03.357 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-29T11:31:03.357 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-29T11:31:03.412 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-29T11:31:03.413 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-29T11:31:03.556 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-29T11:31:03.655 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:03.793 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:03.941 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:03.942 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:31:03.943 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:03.943 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 85 k 2024-03-29T11:31:03.943 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 231 k 2024-03-29T11:31:03.943 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:31:04.090 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:04.361 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 204 kB/s | 85 kB 00:00 2024-03-29T11:31:04.361 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:04.362 INFO:teuthology.orchestra.run.smithi071.stdout:Total 203 kB/s | 85 kB 00:00 2024-03-29T11:31:04.362 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:31:04.369 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:31:04.369 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:31:04.406 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:04.406 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:04.406 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-29T11:31:04.406 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:04.406 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:04.406 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-29T11:31:04.424 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:31:04.424 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:31:04.560 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:31:04.659 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:04.663 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:05.044 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:05.345 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:05.345 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:31:05.345 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:05.345 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:05.345 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:05.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-mirror 2024-03-29T11:31:06.213 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:06.369 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repo Size 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:06.370 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:31:06.371 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:06.371 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 3.1 M 2024-03-29T11:31:06.371 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 13 M 2024-03-29T11:31:06.371 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:31:07.263 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.5 MB/s | 3.1 MB 00:00 2024-03-29T11:31:07.264 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:07.264 INFO:teuthology.orchestra.run.smithi071.stdout:Total 3.5 MB/s | 3.1 MB 00:00 2024-03-29T11:31:07.264 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:31:07.273 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:31:07.274 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:31:07.348 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:31:07.348 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:31:07.775 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:31:07.839 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:07.864 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:07.864 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:07.864 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:31:07.864 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:07.864 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:31:07.864 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:09.004 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:09.004 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:09.004 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:31:09.004 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:09.004 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:09.005 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:09.253 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install rbd-nbd 2024-03-29T11:31:09.834 INFO:teuthology.orchestra.run.smithi071.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:29:13 AM UTC. 2024-03-29T11:31:09.995 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-29T11:31:09.996 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:09.996 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-29T11:31:09.996 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:09.996 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-29T11:31:09.996 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout:Install 1 Package 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 171 k 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 494 k 2024-03-29T11:31:09.997 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-29T11:31:10.675 INFO:teuthology.orchestra.run.smithi071.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 253 kB/s | 171 kB 00:00 2024-03-29T11:31:10.676 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:10.676 INFO:teuthology.orchestra.run.smithi071.stdout:Total 252 kB/s | 171 kB 00:00 2024-03-29T11:31:10.676 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-29T11:31:10.684 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-29T11:31:10.684 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-29T11:31:10.740 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-29T11:31:10.741 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-29T11:31:10.887 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-29T11:31:10.983 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:11.375 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:11.661 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:31:11.661 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:11.662 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-29T11:31:11.662 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:11.662 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:11.662 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-29T11:31:11.910 DEBUG:teuthology.parallel:result is None 2024-03-29T11:31:11.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:12.055 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:31:12.075 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:12.076 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:12.076 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:31:12.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:31:12.219 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:31:12.244 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:12.244 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:31:12.244 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:31:12.246 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:31:12.247 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:12.247 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:31:12.274 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:12.274 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:31:12.315 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:31:12.315 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:12.316 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:31:12.342 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:31:12.409 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:12.409 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:31:12.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:31:12.508 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:31:12.508 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:12.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:31:12.535 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:31:12.622 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:12.622 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:31:12.651 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:31:12.720 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:31:12.721 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:12.721 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:31:12.748 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:31:12.815 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:12.815 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:31:12.844 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:31:12.912 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:31:12.998 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:31:12.998 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /var/log/ceph 2024-03-29T11:31:13.001 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-03-29T11:31:13.028 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:31:13.028 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:31:13.068 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:31:13.095 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:31:13.095 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:31:13.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:31:13.172 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:31:13.172 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:31:13.172 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622716', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi071.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsw3D26UudSyhWAXvfU35it+Nkk+lU6EapXFrTTx4yL3V7UtRqrKhirIsmYe1pOnNZZ/beEEY1Wztdgab8ahDU=', 'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQvsr5fyqktjrh12Z71e6Rxc5GaVazbKAgCg4Zym6vhpl4LSoM7JfyI6EhK4lynfYfgcQvapZ4K0zZRCmK1gb8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:31:13.173 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:31:13.203 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:31:13.218 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:31:13.258 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:31:13.284 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:13.285 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:31:13.338 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:31:13.338 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 11:31:08.118794154 +0000 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 11:28:18.677310803 +0000 2024-03-29T11:31:13.393 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 11:28:18.677310803 +0000 2024-03-29T11:31:13.394 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 11:28:18.677310803 +0000 2024-03-29T11:31:13.394 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:31:13.459 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T11:31:13.459 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T11:31:13.459 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000256996 s, 2.0 MB/s 2024-03-29T11:31:13.460 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:31:13.516 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 11:31:08.119794137 +0000 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 11:28:19.042304324 +0000 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 11:28:19.042304324 +0000 2024-03-29T11:31:13.572 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 11:28:19.042304324 +0000 2024-03-29T11:31:13.573 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:31:13.637 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T11:31:13.638 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T11:31:13.638 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000233547 s, 2.2 MB/s 2024-03-29T11:31:13.639 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:31:13.695 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-03-29T11:31:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:31:13.750 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:13.750 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:31:13.750 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:13.751 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:13.751 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 11:31:08.119794137 +0000 2024-03-29T11:31:13.751 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 11:28:19.579294791 +0000 2024-03-29T11:31:13.751 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 11:28:19.579294791 +0000 2024-03-29T11:31:13.751 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 11:28:19.579294791 +0000 2024-03-29T11:31:13.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:31:13.815 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T11:31:13.816 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T11:31:13.816 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000182576 s, 2.8 MB/s 2024-03-29T11:31:13.817 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:31:13.872 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-03-29T11:31:13.927 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:31:13.927 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:13.927 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:31:13.927 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:13.927 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:13.928 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-29 11:31:08.119794137 +0000 2024-03-29T11:31:13.928 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-29 11:28:19.925288648 +0000 2024-03-29T11:31:13.928 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-29 11:28:19.925288648 +0000 2024-03-29T11:31:13.928 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-29 11:28:19.925288648 +0000 2024-03-29T11:31:13.928 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:31:13.993 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-29T11:31:13.994 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-29T11:31:13.994 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000281642 s, 1.8 MB/s 2024-03-29T11:31:13.995 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:31:14.050 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:14.050 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:14.050 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:31:14.068 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:31:14.068 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-03-29T11:31:14.124 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:31:14.124 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-29 11:31:01.247802762 +0000 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-29 11:28:21.347596219 +0000 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-29 11:28:21.347596219 +0000 2024-03-29T11:31:14.125 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-29 11:28:21.347596219 +0000 2024-03-29T11:31:14.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:31:14.192 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-29T11:31:14.192 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-29T11:31:14.192 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000271166 s, 1.9 MB/s 2024-03-29T11:31:14.194 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:31:14.251 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-03-29T11:31:14.307 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:31:14.307 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:14.307 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:31:14.307 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:14.307 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:14.307 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-29 11:31:01.247802762 +0000 2024-03-29T11:31:14.308 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-29 11:28:21.686589708 +0000 2024-03-29T11:31:14.308 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-29 11:28:21.686589708 +0000 2024-03-29T11:31:14.308 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-29 11:28:21.686589708 +0000 2024-03-29T11:31:14.308 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:31:14.374 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-29T11:31:14.374 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-29T11:31:14.375 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000193354 s, 2.6 MB/s 2024-03-29T11:31:14.376 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:31:14.433 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-29 11:31:01.248802743 +0000 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-29 11:28:22.044582832 +0000 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-29 11:28:22.045582813 +0000 2024-03-29T11:31:14.489 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-29 11:28:22.044582832 +0000 2024-03-29T11:31:14.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:31:14.555 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-29T11:31:14.556 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-29T11:31:14.556 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000203772 s, 2.5 MB/s 2024-03-29T11:31:14.557 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:31:14.613 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-03-29T11:31:14.668 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:31:14.668 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:31:14.668 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:31:14.668 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:31:14.669 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:31:14.669 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-29 11:31:01.248802743 +0000 2024-03-29T11:31:14.669 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-29 11:28:22.391576168 +0000 2024-03-29T11:31:14.669 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-29 11:28:22.391576168 +0000 2024-03-29T11:31:14.669 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-29 11:28:22.391576168 +0000 2024-03-29T11:31:14.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:31:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-29T11:31:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-29T11:31:14.734 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000167083 s, 3.1 MB/s 2024-03-29T11:31:14.735 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:31:14.791 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:14.791 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:31:14.791 INFO:tasks.ceph:Generating config... 2024-03-29T11:31:14.792 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:31:14.792 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:31:14.792 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:31:14.792 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:31:14.793 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:31:14.793 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:31:14.793 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:31:14.793 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:31:14.793 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:31:14.793 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:31:14.793 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:31:14.794 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:31:14.794 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:31:14.794 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:31:14.794 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:31:14.795 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:31:14.796 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:31:14.796 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:31:14.847 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:31:14.850 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:31:14.894 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:31:14.920 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.71:6789]'), ('mon.c', '[v1:172.21.15.71:6790]'), ('mon.b', '[v1:172.21.15.81:6789]')] 2024-03-29T11:31:14.921 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.81:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:31:14.921 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:14.922 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:31:14.975 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.71:6789]' --addv c '[v1:172.21.15.71:6790]' --addv b '[v1:172.21.15.81:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: generated fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:setting min_mon_release = quincy 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:epoch 0 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:min_mon_release 17 (quincy) 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:election_strategy: 1 2024-03-29T11:31:15.060 INFO:teuthology.orchestra.run.smithi071.stdout:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:15.061 INFO:teuthology.orchestra.run.smithi071.stdout:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:15.061 INFO:teuthology.orchestra.run.smithi071.stdout:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:15.061 INFO:teuthology.orchestra.run.smithi071.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:31:15.062 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:31:15.076 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4e4d801d-5e35-484d-91ad-f481aed4a661... 2024-03-29T11:31:15.077 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:31:15.119 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:31:15.166 INFO:teuthology.orchestra.run.smithi071.stdout:[global] 2024-03-29T11:31:15.166 INFO:teuthology.orchestra.run.smithi071.stdout: chdir = "" 2024-03-29T11:31:15.166 INFO:teuthology.orchestra.run.smithi071.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: auth supported = cephx 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: filestore xattr use omap = true 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: mon clock drift allowed = 1.000 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: osd crush chooseleaf type = 0 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: auth debug = true 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on old message = true 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: ms die on bug = true 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:31:15.167 INFO:teuthology.orchestra.run.smithi071.stdout: mon pg warn max object skew = 0 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default size = 2 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow primary affinity = true 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd allow pg remap = true 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on no sortbitwise = false 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on too few osds = false 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:31:15.168 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon_allow_pool_size_one = true 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: osd default data pool replay window = 5 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon allow pool delete = true 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon cluster log file level = debug 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: debug asserts on shutdown = true 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon health detail to clog = false 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon host = "[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.81:6789]" 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: lockdep = True 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon client directed command retry = 5 2024-03-29T11:31:15.169 INFO:teuthology.orchestra.run.smithi071.stdout: mon election default strategy = 1 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: ms bind msgr2 = False 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: ms inject socket failures = 5000 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: ms type = async 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: fsid = 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout:[osd] 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: osd journal size = 100 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub load threshold = 5.0 2024-03-29T11:31:15.170 INFO:teuthology.orchestra.run.smithi071.stdout: osd scrub max interval = 600 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd recover clone overlap = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd recovery max chunk = 1048576 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug shutdown = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug op order = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify stray on activate = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug trim objects = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd open classes on start = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug pg log writeout = true 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.171 INFO:teuthology.orchestra.run.smithi071.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: osd map max advance = 10 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: journal zero on create = true 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: filestore apply finisher threads = 3 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: bdev debug aio = true 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug misdirected ops = true 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore block size = 96636764160 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression algorithm = snappy 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore compression mode = aggressive 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore fsck on mount = True 2024-03-29T11:31:15.172 INFO:teuthology.orchestra.run.smithi071.stdout: bluestore zero block detection = True 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluefs = 20 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: debug bluestore = 20 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: debug osd = 20 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: debug rocksdb = 10 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd full ratio = 0.9 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: osd class default list = * 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: osd class load list = * 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: osd client watch timeout = 120 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify cached snaps = True 2024-03-29T11:31:15.173 INFO:teuthology.orchestra.run.smithi071.stdout: osd debug verify missing on start = True 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd mclock override recovery settings = True 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd objectstore = bluestore 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue = debug_random 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd op queue cut off = debug_random 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: osd shutdown pgref assert = True 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr] 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: debug mgr = 20 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:15.174 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mgr/telemetry/nag = false 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout:[mon] 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: debug mon = 20 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: debug paxos = 20 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: debug auth = 20 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mon data avail warn = 5 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mon mgr mkfs grace = 240 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd reporter subtree level = osd 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mon osd prime pg temp = true 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.175 INFO:teuthology.orchestra.run.smithi071.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: # 1m isn't quite enough 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon min osdmap epochs = 50 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune interval = 2 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune min = 15 2024-03-29T11:31:15.176 INFO:teuthology.orchestra.run.smithi071.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: mon scrub interval = 300 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: mon warn on pool no app = False 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: paxos service trim min = 10 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout:[client] 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: rgw cache enabled = true 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable ops log = true 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: rgw enable usage log = true 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: debug monc = 20 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: debug ms = 1 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: debug objecter = 20 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout: debug rados = 20 2024-03-29T11:31:15.177 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.a] 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.c] 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi071.stdout:[mon.b] 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-03-29T11:31:15.178 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:31:15.179 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-03-29T11:31:15.180 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "[v1:172.21.15.71:6789],[v1:172.21.15.71:6790],[v1:172.21.15.81:6789]" 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: lockdep = True 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: mon election default strategy = 1 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: ms bind msgr2 = False 2024-03-29T11:31:15.181 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 5000 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: ms type = async 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-03-29T11:31:15.182 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.183 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore compression algorithm = snappy 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore compression mode = aggressive 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore zero block detection = True 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 20 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 20 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-03-29T11:31:15.184 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd class default list = * 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd class load list = * 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd client watch timeout = 120 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify cached snaps = True 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify missing on start = True 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock override recovery settings = True 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue = debug_random 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue cut off = debug_random 2024-03-29T11:31:15.185 INFO:teuthology.orchestra.run.smithi081.stdout: osd shutdown pgref assert = True 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-03-29T11:31:15.186 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:31:15.187 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon min osdmap epochs = 50 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune interval = 2 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune min = 15 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon scrub interval = 300 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on pool no app = False 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: paxos service trim min = 10 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-03-29T11:31:15.188 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: debug objecter = 20 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout: debug rados = 20 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-03-29T11:31:15.189 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-03-29T11:31:15.191 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:31:15.191 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:31:15.286 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:31:15.286 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:15.286 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:31:15.301 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:15.301 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:31:15.357 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:31:15.357 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:15.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:31:15.357 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:31:15.435 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:15.435 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:15.490 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:31:15.490 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:15.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:31:15.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:31:15.530 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:15.530 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:15.585 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:31:15.585 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:31:15.585 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:31:15.640 INFO:teuthology.orchestra.run.smithi071.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:31:15.643 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:31:15.703 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:31:15.706 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:31:15.706 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:31:15.706 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:31:15.749 INFO:teuthology.orchestra.run.smithi071.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:31:15.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:31:15.806 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:31:15.822 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:31:15.822 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi071.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:31:15.822 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:15.848 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:15.849 INFO:tasks.ceph:role: osd.0 2024-03-29T11:31:15.849 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:31:15.849 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:31:15.922 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:15.923 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:16.434 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T11:31:16.436 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T11:31:16.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:16.548 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:16.582 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:16.650 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:16.650 INFO:tasks.ceph:role: osd.1 2024-03-29T11:31:16.650 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:31:16.650 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:16.725 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:16.726 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:16.726 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:17.186 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T11:31:17.188 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T11:31:17.188 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:17.292 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:17.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:17.390 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:17.392 INFO:tasks.ceph:role: osd.2 2024-03-29T11:31:17.392 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:31:17.393 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:17.465 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:17.466 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:17.466 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:17.466 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:17.949 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T11:31:17.950 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T11:31:17.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:18.096 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:18.128 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:18.195 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:18.195 INFO:tasks.ceph:role: osd.3 2024-03-29T11:31:18.195 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:31:18.196 DEBUG:teuthology.orchestra.run.smithi071:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:31:18.270 INFO:teuthology.orchestra.run.smithi071.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:18.270 INFO:teuthology.orchestra.run.smithi071.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:18.270 INFO:teuthology.orchestra.run.smithi071.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:18.271 INFO:teuthology.orchestra.run.smithi071.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:18.271 INFO:teuthology.orchestra.run.smithi071.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:18.271 INFO:teuthology.orchestra.run.smithi071.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:18.271 INFO:teuthology.orchestra.run.smithi071.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:18.796 INFO:teuthology.orchestra.run.smithi071.stdout:Discarding blocks...Done. 2024-03-29T11:31:18.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi071.front.sepia.ceph.com -o noatime 2024-03-29T11:31:18.798 DEBUG:teuthology.orchestra.run.smithi071:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:18.921 DEBUG:teuthology.orchestra.run.smithi071:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:18.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:19.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:19.059+0000 7f1eb5bbd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:31:19.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:19.060+0000 7f1eb5bbd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:31:19.053 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:19.060+0000 7f1eb5bbd640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:31:19.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:19.060+0000 7f1eb5bbd640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:31:19.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:19.060+0000 7f1eb5bbd640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:31:19.054 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:19.060+0000 7f1eb5bbd640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:31:21.899 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:21.929 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:22.026+0000 7fb292a0f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:31:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:22.026+0000 7fb292a0f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:31:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:22.026+0000 7fb292a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:31:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:22.026+0000 7fb292a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:31:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:22.026+0000 7fb292a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:31:22.020 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:22.026+0000 7fb292a0f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:31:24.862 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:24.893 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:24.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:24.992+0000 7f605770d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:31:24.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:24.992+0000 7f605770d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:31:24.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:24.992+0000 7f605770d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:31:24.986 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:24.992+0000 7f605770d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:31:24.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:24.992+0000 7f605770d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:31:24.987 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:24.993+0000 7f605770d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:31:27.830 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:27.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:27.957+0000 7f77d7a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:31:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:27.957+0000 7f77d7a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:31:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:27.957+0000 7f77d7a13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:31:27.951 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:27.957+0000 7f77d7a13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:31:27.952 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:27.957+0000 7f77d7a13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:31:27.952 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:27.958+0000 7f77d7a13640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:31:30.817 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:30.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:30.880 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:30.880 INFO:tasks.ceph:role: osd.4 2024-03-29T11:31:30.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:31:30.880 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:31:30.958 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:30.958 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:30.962 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:31.469 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-29T11:31:31.471 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-29T11:31:31.472 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:31.589 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:31.625 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:31.695 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:31.695 INFO:tasks.ceph:role: osd.5 2024-03-29T11:31:31.695 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:31:31.695 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:31.771 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:31.772 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:31.772 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:31.772 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:32.234 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-29T11:31:32.236 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-29T11:31:32.236 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:32.324 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:32.355 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:32.422 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:32.422 INFO:tasks.ceph:role: osd.6 2024-03-29T11:31:32.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:31:32.423 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:32.497 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:32.984 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-29T11:31:32.985 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-29T11:31:32.986 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:33.127 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:33.160 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:33.227 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:31:33.227 INFO:tasks.ceph:role: osd.7 2024-03-29T11:31:33.227 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:31:33.227 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:31:33.301 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:31:33.302 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:31:33.833 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-29T11:31:33.835 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-29T11:31:33.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:33.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:34.006 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:34.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:34.109+0000 7f5cd6b91640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:34.109+0000 7f5cd6b91640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:34.109+0000 7f5cd6b91640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:34.109+0000 7f5cd6b91640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:34.109+0000 7f5cd6b91640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:31:34.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:34.109+0000 7f5cd6b91640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:31:36.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:36.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:37.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:37.081+0000 7ff02b60c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:31:37.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:37.081+0000 7ff02b60c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:31:37.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:37.081+0000 7ff02b60c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:31:37.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:37.081+0000 7ff02b60c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:31:37.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:37.081+0000 7ff02b60c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:31:37.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:37.082+0000 7ff02b60c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:31:39.921 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:39.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:40.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:40.050+0000 7f6e7d4d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:31:40.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:40.051+0000 7f6e7d4d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:31:40.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:40.051+0000 7f6e7d4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:31:40.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:40.051+0000 7f6e7d4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:31:40.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:40.051+0000 7f6e7d4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:31:40.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:40.051+0000 7f6e7d4d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:31:42.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:42.933 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:43.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:43.031+0000 7effd892a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:31:43.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:43.031+0000 7effd892a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:31:43.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:43.031+0000 7effd892a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:31:43.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:43.031+0000 7effd892a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:31:43.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:43.031+0000 7effd892a640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:31:43.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:43.032+0000 7effd892a640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:31:45.880 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:45.911 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:31:45.911 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:45.912 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:31:45.944 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:45.944 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:31:46.014 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:46.014 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:31:46.083 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:46.084 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:31:46.153 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:46.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:31:46.221 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:46.222 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:31:46.253 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:46.253 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:31:46.323 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:46.323 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:31:46.393 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:46.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:31:46.462 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:46.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:31:46.532 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:46.532 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:31:46.549 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:46.549 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:31:46.588 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:31:46.588 DEBUG:teuthology.orchestra.run.smithi071:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:31:46.591 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:31:46.631 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.y] 2024-03-29T11:31:46.631 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCDpgZmbkqdJhAAD3DSIq29fOiuzzpJX8Tr3A== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.0] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCHpgZmqcGeAxAAaV67NL+73jEYrAVwSO6aYg== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.1] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCKpgZmHEicARAA6TogI2Kfw9Qs/OJAR4DX3w== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.2] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCMpgZmrvMxOxAAiM8sfrdIySXteG8Pqp6pLA== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.3] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCPpgZmnlgcORAAuUsvJwDPLgiUscc7EkqenA== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[mgr.x] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCDpgZm/NRhKhAAZFqgwyZ36iFNrZLgCk/UXA== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.4] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCWpgZmy/OOBhAAovKBXdvbsbdLV7/yJOvnuw== 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.5] 2024-03-29T11:31:46.632 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCZpgZmd7TkBBAAOZ94bTNvbgSF96nA67bP0A== 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.6] 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCcpgZmAEMVAxAAtoBlIRIDQ5zpyWzYIXTTTA== 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout:[osd.7] 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCfpgZmC+vtARAANhgdxBZBjjgXdchcJmANAQ== 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout:[client.0] 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCDpgZmN2kXLRAAGZDwCPpQD3CFT/Htu1GusQ== 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-03-29T11:31:46.633 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQCDpgZmj5N/MBAAMJnOt1AN5bSkL7EHh8dfRQ== 2024-03-29T11:31:46.655 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2024-03-29T11:31:46.655 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCDpgZmbkqdJhAAD3DSIq29fOiuzzpJX8Tr3A== 2024-03-29T11:31:46.655 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-03-29T11:31:46.655 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCHpgZmqcGeAxAAaV67NL+73jEYrAVwSO6aYg== 2024-03-29T11:31:46.655 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCKpgZmHEicARAA6TogI2Kfw9Qs/OJAR4DX3w== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCMpgZmrvMxOxAAiM8sfrdIySXteG8Pqp6pLA== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCPpgZmnlgcORAAuUsvJwDPLgiUscc7EkqenA== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCDpgZm/NRhKhAAZFqgwyZ36iFNrZLgCk/UXA== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCWpgZmy/OOBhAAovKBXdvbsbdLV7/yJOvnuw== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCZpgZmd7TkBBAAOZ94bTNvbgSF96nA67bP0A== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCcpgZmAEMVAxAAtoBlIRIDQ5zpyWzYIXTTTA== 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-03-29T11:31:46.656 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCfpgZmC+vtARAANhgdxBZBjjgXdchcJmANAQ== 2024-03-29T11:31:46.657 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-03-29T11:31:46.657 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCDpgZmN2kXLRAAGZDwCPpQD3CFT/Htu1GusQ== 2024-03-29T11:31:46.657 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2024-03-29T11:31:46.657 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCDpgZmj5N/MBAAMJnOt1AN5bSkL7EHh8dfRQ== 2024-03-29T11:31:46.658 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:31:46.678 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:31:46.752 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.755 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.861 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.863 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.914 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:46.968 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:31:46.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:31:47.021 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.024 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.076 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.079 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.131 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.184 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.187 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:31:47.239 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:31:47.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:31:47.296 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:31:47.298 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:31:47.352 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:31:47.353 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:31:47.380 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:31:47.476 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:31:47.504 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:31:47.571 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:31:47.667 DEBUG:teuthology.orchestra.run.smithi071:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:31:47.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:31:47.723 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:31:47.822 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:31:47.851 DEBUG:teuthology.orchestra.run.smithi071:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:47.853 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:31:47.906 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:31:47.906 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:31:47.907 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:31:47.909 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:31:47.909 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:31:47.909 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:31:47.911 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:31:47.911 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:31:47.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:31:47.948 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:31:47.948 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:31:47.949 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:31:47.949 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:31:47.951 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:31:47.951 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:31:47.951 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:31:47.953 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:31:47.953 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:31:47.953 DEBUG:teuthology.orchestra.run.smithi071:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:31:48.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.061+0000 7f63dd0c1640 10 monclient: get_monmap_and_config 2024-03-29T11:31:48.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.061+0000 7f63dd0c1640 10 monclient: build_initial_monmap 2024-03-29T11:31:48.055 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.061+0000 7f63dd0c1640 10 monclient: monmap: 2024-03-29T11:31:48.055 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:31:48.055 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:48.056 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:48.062098+0000 2024-03-29T11:31:48.056 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:48.062098+0000 2024-03-29T11:31:48.056 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:48.056 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:48.056 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:48.056 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:48.057 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:48.057 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:48.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.061+0000 7f63dd0c1640 1 Processor -- start 2024-03-29T11:31:48.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.061+0000 7f63dd0c1640 1 -- start start 2024-03-29T11:31:48.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.061+0000 7f63dd0c1640 10 monclient: init 2024-03-29T11:31:48.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f63d809ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f63d809c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f63d80b4360 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d809dec0 con 0x7f63d80b4360 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d809da30 con 0x7f63d809ac40 2024-03-29T11:31:48.059 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80b39c0 con 0x7f63d809c870 2024-03-29T11:31:48.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:48.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63dd0c1640 10 monclient(hunting): authenticate will time out at 762.216412s 2024-03-29T11:31:48.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63d659c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f63d809c870 0x7f63d813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:41670/0 (socket says 172.21.15.71:41670) 2024-03-29T11:31:48.060 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:48.062+0000 7f63d659c640 1 -- 172.21.15.71:0/3791151132 learned_addr learned my addr 172.21.15.71:0/3791151132 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:31:51.057 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): tick 2024-03-29T11:31:51.057 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:31:51.057 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 monclient(hunting): continuing hunt 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6790/0 conn(0x7f63d809c870 legacy=0x7f63d813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.81:6789/0 conn(0x7f63d809ac40 legacy=0x7f63d80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6789/0 conn(0x7f63d80b4360 legacy=0x7f63d814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): picked mon.noname-a con 0x7f63c40036e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): picked mon.noname-b con 0x7f63c40072e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): picked mon.noname-c con 0x7f63d809ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80b5160 con 0x7f63c40036e0 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c400f820 con 0x7f63d809ac40 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4010a80 con 0x7f63c40072e0 2024-03-29T11:31:51.058 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:51.063+0000 7f63d4d99640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.230+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153198976 0 0) 0x7f63d80b5160 con 0x7f63c40036e0 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.230+0000 7f63d5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.230+0000 7f63d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.230+0000 7f63d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.230+0000 7f63d5d9b640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.230+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0003520 con 0x7f63c40036e0 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282844802 0 0) 0x7f63c4010a80 con 0x7f63c40072e0 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.227 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d80b5160 con 0x7f63c40072e0 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d659c640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6790/0 conn(0x7f63c40072e0 legacy=0x7f63c4009730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d659c640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6790/0 conn(0x7f63c40072e0 legacy=0x7f63c4009730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d659c640 1 --1- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6790/0 conn(0x7f63c40072e0 0x7f63c4009730 :-1 s=OPENED pgs=12 cs=1 l=1).handle_message read tag failed 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d659c640 1 --1- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6790/0 conn(0x7f63c40072e0 0x7f63c4009730 :-1 s=OPENED pgs=12 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.71:6790/0 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290752670 0 0) 0x7f63c400f820 con 0x7f63d809ac40 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:31:53.228 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4010a80 con 0x7f63d809ac40 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287660475 0 0) 0x7f63c4010a80 con 0x7f63d809ac40 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6790/0 conn(0x7f63c40072e0 legacy=0x7f63c4009730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.71:6789/0 conn(0x7f63c40036e0 legacy=0x7f63c4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4011ce0 con 0x7f63d809ac40 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.229 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f63d00017f0 con 0x7f63d809ac40 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63d5d9b640 10 monclient: dump: 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.230 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63dd0c1640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.231+0000 7f63dd0c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63d0001950 con 0x7f63d809ac40 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63d5d9b640 1 -- 172.21.15.71:0/3791151132 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f63d0001da0 con 0x7f63d809ac40 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63d5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63d5d9b640 10 monclient: dump: 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.231 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63dd0c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:53.232 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.232+0000 7f63dd0c1640 4 set_mon_vals no callback set 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 10 monclient: shutdown 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3791151132 >> v1:172.21.15.81:6789/0 conn(0x7f63d809ac40 legacy=0x7f63c400cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3791151132 shutdown_connections 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3791151132 >> 172.21.15.71:0/3791151132 conn(0x7f63d8066b00 msgr2=0x7f63d8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3791151132 shutdown_connections 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.233+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3791151132 wait complete. 2024-03-29T11:31:53.238 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.234+0000 7f63dd0c1640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.234+0000 7f63dd0c1640 10 monclient: monmap: 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 1 librados: starting msgr at 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 1 librados: starting objecter 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 1 Processor -- start 2024-03-29T11:31:53.239 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 1 -- start start 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 1 librados: setting wanted keys 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 1 librados: calling monclient init 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.235+0000 7f63dd0c1640 10 monclient: init 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f63c40036e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f63c40072e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f63d809ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d81ca550 con 0x7f63c40036e0 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d81d7020 con 0x7f63d809ac40 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80b7960 con 0x7f63c40072e0 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63dd0c1640 10 monclient(hunting): authenticate will time out at 767.390368s 2024-03-29T11:31:53.240 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63d6d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f63c40036e0 0x7f63d81cbaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35654/0 (socket says 172.21.15.71:35654) 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63d6d9d640 1 -- 172.21.15.71:0/3577759520 learned_addr learned my addr 172.21.15.71:0/3577759520 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 client.?.objecter ms_handle_connect 0x7f63c40036e0 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 client.?.objecter ms_handle_connect 0x7f63c40072e0 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 client.?.objecter ms_handle_connect 0x7f63d809ac40 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730808146 0 0) 0x7f63d81ca550 con 0x7f63c40036e0 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c40033f0 con 0x7f63c40036e0 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3092538614 0 0) 0x7f63c40033f0 con 0x7f63c40036e0 2024-03-29T11:31:53.241 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63d81ca550 con 0x7f63c40036e0 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f63cc001990 con 0x7f63c40036e0 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): dump: 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:53.242 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.236+0000 7f63bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 661398387 0 0) 0x7f63d81ca550 con 0x7f63c40036e0 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 >> v1:172.21.15.71:6790/0 conn(0x7f63c40072e0 legacy=0x7f63d81d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.243 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 >> v1:172.21.15.81:6789/0 conn(0x7f63d809ac40 legacy=0x7f63d80b7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 monclient: found mon.a 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d81d6cf0 con 0x7f63c40036e0 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 10 monclient: _renew_subs 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:53.244 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d80b6770 con 0x7f63c40036e0 2024-03-29T11:31:53.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:53.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 10 monclient: _renew_subs 2024-03-29T11:31:53.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:53.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d8136050 con 0x7f63c40036e0 2024-03-29T11:31:53.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63dd0c1640 1 librados: init done 2024-03-29T11:31:53.245 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63cc001c90 con 0x7f63c40036e0 2024-03-29T11:31:53.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f63cc003a40 con 0x7f63c40036e0 2024-03-29T11:31:53.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:53.246 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 monclient: dump: 2024-03-29T11:31:53.246 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.247 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:53.248 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63d559a640 4 set_mon_vals no callback set 2024-03-29T11:31:53.248 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f63cc003fb0 con 0x7f63c40036e0 2024-03-29T11:31:53.248 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3732763146 0 0) 0x7f63cc006700 con 0x7f63c40036e0 2024-03-29T11:31:53.248 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 10 client.4110.objecter ms_dispatch 0x7f63d8000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:31:53.248 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:31:53.248 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:31:53.249 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.237+0000 7f63bf7fe640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.238+0000 7f63dd0c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.238+0000 7f63dd0c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.238+0000 7f63dd0c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.238+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b8005080 con 0x7f63c40036e0 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.241+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f63cc0069e0 con 0x7f63c40036e0 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.241+0000 7f63bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.241+0000 7f63bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:53.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.312+0000 7f63dd0c1640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:31:53.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.312+0000 7f63dd0c1640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:31:53.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.312+0000 7f63dd0c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:53.306 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.312+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f63b8005330 con 0x7f63c40036e0 2024-03-29T11:31:53.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.313+0000 7f63bf7fe640 1 -- 172.21.15.71:0/3577759520 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f63cc005880 con 0x7f63c40036e0 2024-03-29T11:31:53.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.313+0000 7f63bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:31:53.308 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.313+0000 7f63bf7fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:31:53.308 INFO:teuthology.orchestra.run.smithi071.stderr:adjusted tunables profile to default 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.314+0000 7f63dd0c1640 10 librados: watch_flush enter 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.314+0000 7f63dd0c1640 10 librados: watch_flush exit 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.314+0000 7f63dd0c1640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.314+0000 7f63dd0c1640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 10 monclient: shutdown 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 >> v1:172.21.15.71:6789/0 conn(0x7f63c40036e0 legacy=0x7f63d81cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 shutdown_connections 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 >> 172.21.15.71:0/3577759520 conn(0x7f63d8066b00 msgr2=0x7f63d80b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 shutdown_connections 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 1 -- 172.21.15.71:0/3577759520 wait complete. 2024-03-29T11:31:53.310 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:53.315+0000 7f63dd0c1640 1 librados: shutdown 2024-03-29T11:31:53.323 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:31:53.323 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:31:53.323 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:53.323 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:31:53.351 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:53.351 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:31:53.419 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:53.420 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:31:53.488 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-29T11:31:53.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:31:53.557 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:53.557 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:31:53.585 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:53.585 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:31:53.653 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:53.654 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:31:53.696 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:53.701+0000 7f1bfcff1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:31:53.720 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-29T11:31:53.720 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:31:53.721 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:53.725+0000 7f90cbc41040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:31:53.752 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:53.758+0000 7f1bfcff1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:31:53.773 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:53.778+0000 7f90cbc41040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:31:53.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 494f788b-24f0-4777-9c63-fe989d2c07ba 0 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.924+0000 7fba9fa5c640 10 monclient: get_monmap_and_config 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.924+0000 7fba9fa5c640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.924+0000 7fba9fa5c640 10 monclient: monmap: 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:53.925169+0000 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:53.925169+0000 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.919 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.924+0000 7fba9fa5c640 1 Processor -- start 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 1 -- start start 2024-03-29T11:31:53.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient: init 2024-03-29T11:31:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient(hunting): picked mon.noname-a con 0x7fba9809ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:53.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient(hunting): picked mon.noname-b con 0x7fba9809c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient(hunting): picked mon.noname-c con 0x7fba980b4360 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9809dec0 con 0x7fba9809ac40 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba9809da30 con 0x7fba980b4360 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980b39c0 con 0x7fba9809c870 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9fa5c640 10 monclient(hunting): authenticate will time out at 773.188782s 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9dff9640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fba980b4360 0x7fba9814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38522/0 (socket says 172.21.15.81:38522) 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.925+0000 7fba9dff9640 1 -- 172.21.15.81:0/2726767021 learned_addr learned my addr 172.21.15.81:0/2726767021 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:53.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547748586 0 0) 0x7fba9809da30 con 0x7fba980b4360 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba84003520 con 0x7fba980b4360 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645501375 0 0) 0x7fba8c001110 con 0x7fba9809ac40 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba9809da30 con 0x7fba9809ac40 2024-03-29T11:31:53.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145135530 0 0) 0x7fba980b39c0 con 0x7fba9809c870 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8c001110 con 0x7fba9809c870 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2602861742 0 0) 0x7fba84003520 con 0x7fba980b4360 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 >> v1:172.21.15.71:6790/0 conn(0x7fba9809c870 legacy=0x7fba9813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 >> v1:172.21.15.71:6789/0 conn(0x7fba9809ac40 legacy=0x7fba980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 monclient: found mon.noname-c 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba980b5160 con 0x7fba980b4360 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fba90001a10 con 0x7fba980b4360 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: dump: 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:53.933 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba9fa5c640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba9fa5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba90001bf0 con 0x7fba980b4360 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 1 -- 172.21.15.81:0/2726767021 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fba90002000 con 0x7fba980b4360 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba8bfff640 10 monclient: dump: 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.935 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba9fa5c640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:53.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.926+0000 7fba9fa5c640 4 set_mon_vals no callback set 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 10 monclient: shutdown 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 1 -- 172.21.15.81:0/2726767021 >> v1:172.21.15.81:6789/0 conn(0x7fba980b4360 legacy=0x7fba9814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 1 -- 172.21.15.81:0/2726767021 shutdown_connections 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 1 -- 172.21.15.81:0/2726767021 >> 172.21.15.81:0/2726767021 conn(0x7fba98066b00 msgr2=0x7fba98066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 1 -- 172.21.15.81:0/2726767021 shutdown_connections 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 1 -- 172.21.15.81:0/2726767021 wait complete. 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.937 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.927+0000 7fba9fa5c640 10 monclient: monmap: 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 librados: starting msgr at 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 librados: starting objecter 2024-03-29T11:31:53.938 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 Processor -- start 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 -- start start 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 librados: setting wanted keys 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 librados: calling monclient init 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient: init 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient(hunting): picked mon.noname-b con 0x7fba9809ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient(hunting): picked mon.noname-c con 0x7fba98140cc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient(hunting): picked mon.noname-a con 0x7fba98139420 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba98141cf0 con 0x7fba98139420 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980afb40 con 0x7fba9809ac40 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba98139890 con 0x7fba98140cc0 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9fa5c640 10 monclient(hunting): authenticate will time out at 773.192020s 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.928+0000 7fba9d7f8640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fba9809ac40 0x7fba981415e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38528/0 (socket says 172.21.15.81:38528) 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba9d7f8640 1 -- 172.21.15.81:0/4074727206 learned_addr learned my addr 172.21.15.81:0/4074727206 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 client.?.objecter ms_handle_connect 0x7fba9809ac40 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 client.?.objecter ms_handle_connect 0x7fba98140cc0 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 client.?.objecter ms_handle_connect 0x7fba98139420 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488465747 0 0) 0x7fba980afb40 con 0x7fba9809ac40 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:31:53.940 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c004eb0 con 0x7fba9809ac40 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659923905 0 0) 0x7fba98139890 con 0x7fba98140cc0 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba980afb40 con 0x7fba98140cc0 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375316805 0 0) 0x7fba98141cf0 con 0x7fba98139420 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba98139890 con 0x7fba98139420 2024-03-29T11:31:53.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1290815646 0 0) 0x7fba7c004eb0 con 0x7fba9809ac40 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba98141cf0 con 0x7fba9809ac40 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930553339 0 0) 0x7fba98139890 con 0x7fba98139420 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba7c004eb0 con 0x7fba98139420 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fba8c0019a0 con 0x7fba9809ac40 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:53.942 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755024802 0 0) 0x7fba980afb40 con 0x7fba98140cc0 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba98139890 con 0x7fba98140cc0 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fba90001d60 con 0x7fba98139420 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.944 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fba94001830 con 0x7fba98140cc0 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.945 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1034947214 0 0) 0x7fba98141cf0 con 0x7fba9809ac40 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 >> v1:172.21.15.71:6790/0 conn(0x7fba98140cc0 legacy=0x7fba98138d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 >> v1:172.21.15.71:6789/0 conn(0x7fba98139420 legacy=0x7fba980affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 monclient: found mon.b 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98142b90 con 0x7fba9809ac40 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba9fa5c640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba9fa5c640 10 monclient: _renew_subs 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba9fa5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba980b06e0 con 0x7fba9809ac40 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba8c001c80 con 0x7fba9809ac40 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fba8c001b00 con 0x7fba9809ac40 2024-03-29T11:31:53.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 10 monclient: dump: 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:53.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fba8c003e70 con 0x7fba9809ac40 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba9fa5c640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba9fa5c640 10 monclient: _renew_subs 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.929+0000 7fba8b7fe640 4 set_mon_vals no callback set 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba9fa5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba980b0b70 con 0x7fba9809ac40 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba9fa5c640 1 librados: init done 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3218683292 0 0) 0x7fba8c0057c0 con 0x7fba9809ac40 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba89ffb640 10 client.4111.objecter ms_dispatch 0x7fba98000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba89ffb640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba89ffb640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.930+0000 7fba89ffb640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.931+0000 7fba9fa5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.931+0000 7fba9fa5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.931+0000 7fba9fa5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.931+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba981430b0 con 0x7fba9809ac40 2024-03-29T11:31:53.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.933+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fba981430b0 con 0x7fba9809ac40 2024-03-29T11:31:53.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.933+0000 7fba89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:53.933+0000 7fba89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:53.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.005+0000 7fba9fa5c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "494f788b-24f0-4777-9c63-fe989d2c07ba", "id": 0}] 2024-03-29T11:31:53.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.005+0000 7fba9fa5c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "494f788b-24f0-4777-9c63-fe989d2c07ba", "id": 0}] 2024-03-29T11:31:53.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.005+0000 7fba9fa5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:53.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.005+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "494f788b-24f0-4777-9c63-fe989d2c07ba", "id": 0} v 0) v1 -- 0x7fba9804f730 con 0x7fba9809ac40 2024-03-29T11:31:54.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.010+0000 7fba89ffb640 1 -- 172.21.15.81:0/4074727206 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "494f788b-24f0-4777-9c63-fe989d2c07ba", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3823807093 0 732707515) 0x7fba8c005aa0 con 0x7fba9809ac40 2024-03-29T11:31:54.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.010+0000 7fba89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "494f788b-24f0-4777-9c63-fe989d2c07ba", "id": 0}] 2024-03-29T11:31:54.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.010+0000 7fba89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:54.005 INFO:teuthology.orchestra.run.smithi081.stdout:0 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 10 librados: watch_flush enter 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 10 librados: watch_flush exit 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 10 client.4111.objecter successfully canceled tick 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 10 monclient: shutdown 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.011+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 >> v1:172.21.15.81:6789/0 conn(0x7fba9809ac40 legacy=0x7fba981415e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.012+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 shutdown_connections 2024-03-29T11:31:54.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.012+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 >> 172.21.15.81:0/4074727206 conn(0x7fba98066b00 msgr2=0x7fba9813a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.012+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 shutdown_connections 2024-03-29T11:31:54.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.012+0000 7fba9fa5c640 1 -- 172.21.15.81:0/4074727206 wait complete. 2024-03-29T11:31:54.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.012+0000 7fba9fa5c640 1 librados: shutdown 2024-03-29T11:31:54.017 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new e16a4a37-aefb-458a-b2d9-367c94929241 1 2024-03-29T11:31:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.121+0000 7f8f6c3a8640 10 monclient: get_monmap_and_config 2024-03-29T11:31:54.116 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.121+0000 7f8f6c3a8640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.121+0000 7f8f6c3a8640 10 monclient: monmap: 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:54.122732+0000 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:54.122732+0000 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.122+0000 7f8f6c3a8640 1 Processor -- start 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.122+0000 7f8f6c3a8640 1 -- start start 2024-03-29T11:31:54.117 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.122+0000 7f8f6c3a8640 10 monclient: init 2024-03-29T11:31:54.127 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f6409ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f6409c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f640b4360 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6409dec0 con 0x7f8f6409ac40 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6409da30 con 0x7f8f640b4360 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640b39c0 con 0x7f8f6409c870 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient(hunting): authenticate will time out at 773.386263s 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6a945640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8f640b4360 0x7f8f6414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38532/0 (socket says 172.21.15.81:38532) 2024-03-29T11:31:54.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6a945640 1 -- 172.21.15.81:0/3332381253 learned_addr learned my addr 172.21.15.81:0/3332381253 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226121506 0 0) 0x7f8f6409da30 con 0x7f8f640b4360 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f50003520 con 0x7f8f640b4360 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795644230 0 0) 0x7f8f54001110 con 0x7f8f6409ac40 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6409da30 con 0x7f8f6409ac40 2024-03-29T11:31:54.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150516741 0 0) 0x7f8f640b39c0 con 0x7f8f6409c870 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f54001110 con 0x7f8f6409c870 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 704290043 0 0) 0x7f8f50003520 con 0x7f8f640b4360 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 >> v1:172.21.15.71:6790/0 conn(0x7f8f6409c870 legacy=0x7f8f6413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 >> v1:172.21.15.71:6789/0 conn(0x7f8f6409ac40 legacy=0x7f8f640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 monclient: found mon.noname-c 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f640b5160 con 0x7f8f640b4360 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8f5c001a10 con 0x7f8f640b4360 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: dump: 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.131 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f5c001b70 con 0x7f8f640b4360 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 1 -- 172.21.15.81:0/3332381253 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8f5c001f80 con 0x7f8f640b4360 2024-03-29T11:31:54.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f6c3a8640 4 set_mon_vals no callback set 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.123+0000 7f8f69142640 10 monclient: dump: 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.133 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 10 monclient: shutdown 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3332381253 >> v1:172.21.15.81:6789/0 conn(0x7f8f640b4360 legacy=0x7f8f6414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3332381253 shutdown_connections 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3332381253 >> 172.21.15.81:0/3332381253 conn(0x7f8f64066b00 msgr2=0x7f8f64066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3332381253 shutdown_connections 2024-03-29T11:31:54.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.124+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3332381253 wait complete. 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 10 monclient: monmap: 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:54.135 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 1 librados: starting msgr at 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 1 librados: starting objecter 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 1 Processor -- start 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 1 -- start start 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 1 librados: setting wanted keys 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 1 librados: calling monclient init 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.125+0000 7f8f6c3a8640 10 monclient: init 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:54.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f8f6409ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f8f641d8650 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f8f641db840 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f641d7a60 con 0x7f8f6409ac40 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f64142ff0 con 0x7f8f641d8650 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f641dbc70 con 0x7f8f641db840 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f6c3a8640 10 monclient(hunting): authenticate will time out at 773.389323s 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f69943640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8f641d8650 0x7f8f641db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38538/0 (socket says 172.21.15.81:38538) 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f69943640 1 -- 172.21.15.81:0/3082803579 learned_addr learned my addr 172.21.15.81:0/3082803579 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 client.?.objecter ms_handle_connect 0x7f8f641d8650 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 client.?.objecter ms_handle_connect 0x7f8f6409ac40 2024-03-29T11:31:54.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342885070 0 0) 0x7f8f64142ff0 con 0x7f8f641d8650 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f48004eb0 con 0x7f8f641d8650 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 client.?.objecter ms_handle_connect 0x7f8f641db840 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276040895 0 0) 0x7f8f641d7a60 con 0x7f8f6409ac40 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f64142ff0 con 0x7f8f6409ac40 2024-03-29T11:31:54.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3466343836 0 0) 0x7f8f48004eb0 con 0x7f8f641d8650 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f641d7a60 con 0x7f8f641d8650 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123272214 0 0) 0x7f8f641dbc70 con 0x7f8f641db840 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f48004eb0 con 0x7f8f641db840 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8f60001a10 con 0x7f8f641d8650 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): dump: 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.139 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190523163 0 0) 0x7f8f64142ff0 con 0x7f8f6409ac40 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f641dbc70 con 0x7f8f6409ac40 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8f540018e0 con 0x7f8f6409ac40 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 10 monclient(hunting): dump: 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.126+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1613136384 0 0) 0x7f8f641d7a60 con 0x7f8f641d8650 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 >> v1:172.21.15.71:6790/0 conn(0x7f8f641db840 legacy=0x7f8f641e2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 >> v1:172.21.15.71:6789/0 conn(0x7f8f6409ac40 legacy=0x7f8f641d7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 monclient: found mon.b 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f641dbe00 con 0x7f8f641d8650 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 10 monclient: _renew_subs 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64144cc0 con 0x7f8f641d8650 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:54.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 10 monclient: _renew_subs 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f641e3d90 con 0x7f8f641d8650 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f6c3a8640 1 librados: init done 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f60001b70 con 0x7f8f641d8650 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8f60003a60 con 0x7f8f641d8650 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f68941640 4 set_mon_vals no callback set 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 monclient: dump: 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.143 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f60004010 con 0x7f8f641d8650 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1410102217 0 0) 0x7f8f600066f0 con 0x7f8f641d8650 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 10 client.4117.objecter ms_dispatch 0x7f8f64000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.127+0000 7f8f5affd640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.128+0000 7f8f6c3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.128+0000 7f8f6c3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.128+0000 7f8f6c3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.128+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4c005080 con 0x7f8f641d8650 2024-03-29T11:31:54.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.129+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f600068b0 con 0x7f8f641d8650 2024-03-29T11:31:54.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.129+0000 7f8f5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.129+0000 7f8f5affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:54.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.202+0000 7f8f6c3a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e16a4a37-aefb-458a-b2d9-367c94929241", "id": 1}] 2024-03-29T11:31:54.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.202+0000 7f8f6c3a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e16a4a37-aefb-458a-b2d9-367c94929241", "id": 1}] 2024-03-29T11:31:54.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.202+0000 7f8f6c3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.202+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e16a4a37-aefb-458a-b2d9-367c94929241", "id": 1} v 0) v1 -- 0x7f8f4c0058b0 con 0x7f8f641d8650 2024-03-29T11:31:54.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.210+0000 7f8f5affd640 1 -- 172.21.15.81:0/3082803579 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e16a4a37-aefb-458a-b2d9-367c94929241", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4186573031 0 940483276) 0x7f8f600044a0 con 0x7f8f641d8650 2024-03-29T11:31:54.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.210+0000 7f8f5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e16a4a37-aefb-458a-b2d9-367c94929241", "id": 1}] 2024-03-29T11:31:54.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.210+0000 7f8f5affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:54.205 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 10 librados: watch_flush enter 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 10 librados: watch_flush exit 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 10 monclient: shutdown 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 >> v1:172.21.15.81:6789/0 conn(0x7f8f641d8650 legacy=0x7f8f641db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 shutdown_connections 2024-03-29T11:31:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 >> 172.21.15.81:0/3082803579 conn(0x7f8f64066b00 msgr2=0x7f8f6413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 shutdown_connections 2024-03-29T11:31:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 1 -- 172.21.15.81:0/3082803579 wait complete. 2024-03-29T11:31:54.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.211+0000 7f8f6c3a8640 1 librados: shutdown 2024-03-29T11:31:54.216 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 7f9ac3d9-b877-4b68-9183-b33976111dd1 2 2024-03-29T11:31:54.302 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.307+0000 7f1bfcff1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:31:54.305 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.310+0000 7f90cbc41040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.321+0000 7fbf86a63640 10 monclient: get_monmap_and_config 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.321+0000 7fbf86a63640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.321+0000 7fbf86a63640 10 monclient: monmap: 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:54.322612+0000 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:54.322612+0000 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.316 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:54.317 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:54.317 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.322+0000 7fbf86a63640 1 Processor -- start 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.322+0000 7fbf86a63640 1 -- start start 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.322+0000 7fbf86a63640 10 monclient: init 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8009ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient(hunting): picked mon.noname-c con 0x7fbf8009c870 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient(hunting): picked mon.noname-a con 0x7fbf800b4360 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8009dec0 con 0x7fbf800b4360 2024-03-29T11:31:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8009da30 con 0x7fbf8009c870 2024-03-29T11:31:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf800b39c0 con 0x7fbf8009ac40 2024-03-29T11:31:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 10 monclient(hunting): authenticate will time out at 773.586339s 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf77fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fbf8009c870 0x7fbf8013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38550/0 (socket says 172.21.15.81:38550) 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf77fff640 1 -- 172.21.15.81:0/2180729852 learned_addr learned my addr 172.21.15.81:0/2180729852 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163887897 0 0) 0x7fbf8009da30 con 0x7fbf8009c870 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c003520 con 0x7fbf8009c870 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529871548 0 0) 0x7fbf78001490 con 0x7fbf800b4360 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf8009da30 con 0x7fbf800b4360 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521842444 0 0) 0x7fbf800b39c0 con 0x7fbf8009ac40 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf78001490 con 0x7fbf8009ac40 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 600687341 0 0) 0x7fbf6c003520 con 0x7fbf8009c870 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 >> v1:172.21.15.71:6790/0 conn(0x7fbf8009ac40 legacy=0x7fbf800b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 >> v1:172.21.15.71:6789/0 conn(0x7fbf800b4360 legacy=0x7fbf8014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf800b5160 con 0x7fbf8009c870 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fbf68001690 con 0x7fbf8009c870 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 10 monclient: dump: 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.330 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf86a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.323+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf68001870 con 0x7fbf8009c870 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 4 set_mon_vals no callback set 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf7f7fe640 1 -- 172.21.15.81:0/2180729852 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fbf68001c80 con 0x7fbf8009c870 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf7f7fe640 10 monclient: dump: 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 10 monclient: shutdown 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 1 -- 172.21.15.81:0/2180729852 >> v1:172.21.15.81:6789/0 conn(0x7fbf8009c870 legacy=0x7fbf8013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 1 -- 172.21.15.81:0/2180729852 shutdown_connections 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 1 -- 172.21.15.81:0/2180729852 >> 172.21.15.81:0/2180729852 conn(0x7fbf80066b00 msgr2=0x7fbf80066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 1 -- 172.21.15.81:0/2180729852 shutdown_connections 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.324+0000 7fbf86a63640 1 -- 172.21.15.81:0/2180729852 wait complete. 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 10 monclient: monmap: 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:54.333 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 1 librados: starting msgr at 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 1 librados: starting objecter 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 1 Processor -- start 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 1 -- start start 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 1 librados: setting wanted keys 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 1 librados: calling monclient init 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.325+0000 7fbf86a63640 10 monclient: init 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient(hunting): picked mon.noname-b con 0x7fbf8009ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient(hunting): picked mon.noname-c con 0x7fbf800af290 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient(hunting): picked mon.noname-a con 0x7fbf80139400 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf800b1680 con 0x7fbf80139400 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf800b0860 con 0x7fbf8009ac40 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf801399c0 con 0x7fbf800af290 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf86a63640 10 monclient(hunting): authenticate will time out at 773.589367s 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fbf8009ac40 0x7fbf800aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38566/0 (socket says 172.21.15.81:38566) 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7ffff640 1 -- 172.21.15.81:0/3867145803 learned_addr learned my addr 172.21.15.81:0/3867145803 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 client.?.objecter ms_handle_connect 0x7fbf8009ac40 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302412499 0 0) 0x7fbf800b0860 con 0x7fbf8009ac40 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:31:54.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf60004eb0 con 0x7fbf8009ac40 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 client.?.objecter ms_handle_connect 0x7fbf80139400 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 client.?.objecter ms_handle_connect 0x7fbf800af290 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001778303 0 0) 0x7fbf800b1680 con 0x7fbf80139400 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:31:54.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf800b0860 con 0x7fbf80139400 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199064952 0 0) 0x7fbf801399c0 con 0x7fbf800af290 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf800b1680 con 0x7fbf800af290 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1356087172 0 0) 0x7fbf60004eb0 con 0x7fbf8009ac40 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf801399c0 con 0x7fbf8009ac40 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fbf70001a10 con 0x7fbf8009ac40 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.337 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2308405049 0 0) 0x7fbf800b0860 con 0x7fbf80139400 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf60004eb0 con 0x7fbf80139400 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fbf78001a80 con 0x7fbf80139400 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:54.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800130150 0 0) 0x7fbf800b1680 con 0x7fbf800af290 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf800b0860 con 0x7fbf800af290 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039509947 0 0) 0x7fbf801399c0 con 0x7fbf8009ac40 2024-03-29T11:31:54.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 >> v1:172.21.15.71:6790/0 conn(0x7fbf800af290 legacy=0x7fbf80138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 >> v1:172.21.15.71:6789/0 conn(0x7fbf80139400 legacy=0x7fbf801da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 1 monclient: found mon.b 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.326+0000 7fbf7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf801daf50 con 0x7fbf8009ac40 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 monclient: _renew_subs 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf801db1a0 con 0x7fbf8009ac40 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:54.340 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 monclient: _renew_subs 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf801e3af0 con 0x7fbf8009ac40 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 1 librados: init done 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf70001cf0 con 0x7fbf8009ac40 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fbf70003a60 con 0x7fbf8009ac40 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7effd640 4 set_mon_vals no callback set 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 monclient: dump: 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.341 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbf70004010 con 0x7fbf8009ac40 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1063930777 0 0) 0x7fbf700042c0 con 0x7fbf8009ac40 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 10 client.4123.objecter ms_dispatch 0x7fbf80000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf7d7fa640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.342 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.327+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf64005080 con 0x7fbf8009ac40 2024-03-29T11:31:54.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.329+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbf70004480 con 0x7fbf8009ac40 2024-03-29T11:31:54.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.329+0000 7fbf7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.329+0000 7fbf7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:54.359 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.365+0000 7f1bfcff1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:31:54.360 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.365+0000 7f90cbc41040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:31:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.402+0000 7fbf86a63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7f9ac3d9-b877-4b68-9183-b33976111dd1", "id": 2}] 2024-03-29T11:31:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.402+0000 7fbf86a63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7f9ac3d9-b877-4b68-9183-b33976111dd1", "id": 2}] 2024-03-29T11:31:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.402+0000 7fbf86a63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.402+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7f9ac3d9-b877-4b68-9183-b33976111dd1", "id": 2} v 0) v1 -- 0x7fbf640058b0 con 0x7fbf8009ac40 2024-03-29T11:31:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.409+0000 7fbf7d7fa640 1 -- 172.21.15.81:0/3867145803 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f9ac3d9-b877-4b68-9183-b33976111dd1", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1889455749 0 216599125) 0x7fbf70005890 con 0x7fbf8009ac40 2024-03-29T11:31:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.409+0000 7fbf7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7f9ac3d9-b877-4b68-9183-b33976111dd1", "id": 2}] 2024-03-29T11:31:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.409+0000 7fbf7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:54.404 INFO:teuthology.orchestra.run.smithi081.stdout:2 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 10 librados: watch_flush enter 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 10 librados: watch_flush exit 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 10 monclient: shutdown 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 >> v1:172.21.15.81:6789/0 conn(0x7fbf8009ac40 legacy=0x7fbf800aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 shutdown_connections 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.410+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 >> 172.21.15.81:0/3867145803 conn(0x7fbf80066b00 msgr2=0x7fbf8013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.411+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 shutdown_connections 2024-03-29T11:31:54.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.411+0000 7fbf86a63640 1 -- 172.21.15.81:0/3867145803 wait complete. 2024-03-29T11:31:54.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.411+0000 7fbf86a63640 1 librados: shutdown 2024-03-29T11:31:54.407 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.411+0000 7f1bfcff1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:31:54.407 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.410+0000 7f90cbc41040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:31:54.415 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 4cfb3619-8b50-4c13-a7cb-d83b30da8e7f 3 2024-03-29T11:31:54.459 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.464+0000 7f90cbc41040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:31:54.462 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.468+0000 7f1bfcff1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:31:54.501 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.506+0000 7f90cbc41040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:31:54.507 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.513+0000 7f1bfcff1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:31:54.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.521+0000 7fefb236b640 10 monclient: get_monmap_and_config 2024-03-29T11:31:54.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.521+0000 7fefb236b640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.521+0000 7fefb236b640 10 monclient: monmap: 2024-03-29T11:31:54.515 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.515 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:54.522195+0000 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:54.522195+0000 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:54.516 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.521+0000 7fefb236b640 1 Processor -- start 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 1 -- start start 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient: init 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient(hunting): picked mon.noname-a con 0x7fefac09ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient(hunting): picked mon.noname-b con 0x7fefac09c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient(hunting): picked mon.noname-c con 0x7fefac0b4360 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac09dec0 con 0x7fefac09ac40 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac09da30 con 0x7fefac0b4360 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0b39c0 con 0x7fefac09c870 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb236b640 10 monclient(hunting): authenticate will time out at 773.785698s 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb0908640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fefac0b4360 0x7fefac14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38578/0 (socket says 172.21.15.81:38578) 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefb0908640 1 -- 172.21.15.81:0/1953676026 learned_addr learned my addr 172.21.15.81:0/1953676026 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437837269 0 0) 0x7fefac09da30 con 0x7fefac0b4360 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefaaffd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.522+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef94003520 con 0x7fefac0b4360 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262158635 0 0) 0x7fefac0b39c0 con 0x7fefac09c870 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac09da30 con 0x7fefac09c870 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074622593 0 0) 0x7fefa0001040 con 0x7fefac09ac40 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac0b39c0 con 0x7fefac09ac40 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456426120 0 0) 0x7fef94003520 con 0x7fefac0b4360 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 >> v1:172.21.15.71:6790/0 conn(0x7fefac09c870 legacy=0x7fefac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 >> v1:172.21.15.71:6789/0 conn(0x7fefac09ac40 legacy=0x7fefac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 monclient: found mon.noname-c 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac0b5160 con 0x7fefac0b4360 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa4001a10 con 0x7fefac0b4360 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: dump: 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefb236b640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefb236b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa4001bf0 con 0x7fefac0b4360 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 1 -- 172.21.15.81:0/1953676026 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa4002000 con 0x7fefac0b4360 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefaaffd640 10 monclient: dump: 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefb236b640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.523+0000 7fefb236b640 4 set_mon_vals no callback set 2024-03-29T11:31:54.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 10 monclient: shutdown 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 1 -- 172.21.15.81:0/1953676026 >> v1:172.21.15.81:6789/0 conn(0x7fefac0b4360 legacy=0x7fefac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 1 -- 172.21.15.81:0/1953676026 shutdown_connections 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 1 -- 172.21.15.81:0/1953676026 >> 172.21.15.81:0/1953676026 conn(0x7fefac066b00 msgr2=0x7fefac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 1 -- 172.21.15.81:0/1953676026 shutdown_connections 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.524+0000 7fefb236b640 1 -- 172.21.15.81:0/1953676026 wait complete. 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 10 monclient: monmap: 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:54.528 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 1 librados: starting msgr at 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 1 librados: starting objecter 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 1 Processor -- start 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 1 -- start start 2024-03-29T11:31:54.529 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 1 librados: setting wanted keys 2024-03-29T11:31:54.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 1 librados: calling monclient init 2024-03-29T11:31:54.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.525+0000 7fefb236b640 10 monclient: init 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient(hunting): picked mon.noname-a con 0x7fefac09ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient(hunting): picked mon.noname-b con 0x7fefac0aee60 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient(hunting): picked mon.noname-c con 0x7fefac0ae870 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac134a10 con 0x7fefac09ac40 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0b01c0 con 0x7fefac0aee60 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0adaf0 con 0x7fefac0ae870 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefb236b640 10 monclient(hunting): authenticate will time out at 773.789891s 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefab7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fefac0aee60 0x7fefac0b0570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38586/0 (socket says 172.21.15.81:38586) 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefab7fe640 1 -- 172.21.15.81:0/947114651 learned_addr learned my addr 172.21.15.81:0/947114651 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.535 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefa8ff9640 10 client.?.objecter ms_handle_connect 0x7fefac0aee60 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefa8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.526+0000 7fefa8ff9640 10 client.?.objecter ms_handle_connect 0x7fefac09ac40 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 client.?.objecter ms_handle_connect 0x7fefac0ae870 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383735906 0 0) 0x7fefac0b01c0 con 0x7fefac0aee60 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef98004eb0 con 0x7fefac0aee60 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373672893 0 0) 0x7fefac0b39c0 con 0x7fefac09ac40 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.536 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac0b01c0 con 0x7fefac09ac40 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596899270 0 0) 0x7fefac0adaf0 con 0x7fefac0ae870 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac0b39c0 con 0x7fefac0ae870 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3101795982 0 0) 0x7fef98004eb0 con 0x7fefac0aee60 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefac0adaf0 con 0x7fefac0aee60 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fef9c0019b0 con 0x7fefac0aee60 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.537 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2521798542 0 0) 0x7fefac0b01c0 con 0x7fefac09ac40 2024-03-29T11:31:54.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef98004eb0 con 0x7fefac09ac40 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa0001810 con 0x7fefac09ac40 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111663864 0 0) 0x7fefac0b39c0 con 0x7fefac0ae870 2024-03-29T11:31:54.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefac0b01c0 con 0x7fefac0ae870 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa4001f00 con 0x7fefac0ae870 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.540 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3926389455 0 0) 0x7fefac0adaf0 con 0x7fefac0aee60 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 >> v1:172.21.15.71:6790/0 conn(0x7fefac0ae870 legacy=0x7fefac0b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 >> v1:172.21.15.71:6789/0 conn(0x7fefac09ac40 legacy=0x7fefac1d7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 monclient: found mon.b 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac1d7640 con 0x7fefac0aee60 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 10 monclient: _renew_subs 2024-03-29T11:31:54.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac0b1730 con 0x7fefac0aee60 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 10 monclient: _renew_subs 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac1e3e90 con 0x7fefac0aee60 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefb236b640 1 librados: init done 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef9c003610 con 0x7fefac0aee60 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fef9c0039a0 con 0x7fefac0aee60 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefaa7fc640 4 set_mon_vals no callback set 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 monclient: dump: 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.542 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fef9c003f50 con 0x7fefac0aee60 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1417879485 0 0) 0x7fef9c0066e0 con 0x7fefac0aee60 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 10 client.4129.objecter ms_dispatch 0x7fefac000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.527+0000 7fefa8ff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:31:54.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.528+0000 7fefb236b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.528+0000 7fefb236b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.528+0000 7fefb236b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.528+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8c005080 con 0x7fefac0aee60 2024-03-29T11:31:54.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.530+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fef9c004390 con 0x7fefac0aee60 2024-03-29T11:31:54.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.530+0000 7fefa8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.530+0000 7fefa8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:54.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.602+0000 7fefb236b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4cfb3619-8b50-4c13-a7cb-d83b30da8e7f", "id": 3}] 2024-03-29T11:31:54.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.602+0000 7fefb236b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4cfb3619-8b50-4c13-a7cb-d83b30da8e7f", "id": 3}] 2024-03-29T11:31:54.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.602+0000 7fefb236b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.602+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4cfb3619-8b50-4c13-a7cb-d83b30da8e7f", "id": 3} v 0) v1 -- 0x7fef8c0058b0 con 0x7fefac0aee60 2024-03-29T11:31:54.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.610+0000 7fefa8ff9640 1 -- 172.21.15.81:0/947114651 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4cfb3619-8b50-4c13-a7cb-d83b30da8e7f", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3262876190 0 525046306) 0x7fef9c006380 con 0x7fefac0aee60 2024-03-29T11:31:54.604 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.610+0000 7fefa8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4cfb3619-8b50-4c13-a7cb-d83b30da8e7f", "id": 3}] 2024-03-29T11:31:54.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.610+0000 7fefa8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:54.605 INFO:teuthology.orchestra.run.smithi081.stdout:3 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 10 librados: watch_flush enter 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 10 librados: watch_flush exit 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 10 monclient: shutdown 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 >> v1:172.21.15.81:6789/0 conn(0x7fefac0aee60 legacy=0x7fefac0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 shutdown_connections 2024-03-29T11:31:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 >> 172.21.15.81:0/947114651 conn(0x7fefac066b00 msgr2=0x7fefac066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 shutdown_connections 2024-03-29T11:31:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 1 -- 172.21.15.81:0/947114651 wait complete. 2024-03-29T11:31:54.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.611+0000 7fefb236b640 1 librados: shutdown 2024-03-29T11:31:54.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a 4 2024-03-29T11:31:54.683 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.623+0000 7f90cbc41040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:31:54.683 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.636+0000 7f1bfcff1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:31:54.689 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.693+0000 7f90cbc41040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:54.701 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.706+0000 7f1bfcff1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:54.734 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.739+0000 7f90cbc41040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:31:54.749 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.755+0000 7f1bfcff1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.788+0000 7f36d5f82640 10 monclient: get_monmap_and_config 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.788+0000 7f36d5f82640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.788+0000 7f36d5f82640 10 monclient: monmap: 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:54.789353+0000 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:54.789353+0000 2024-03-29T11:31:54.783 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:54.784 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:54.784 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.792 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.791+0000 7f90cbc41040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 1 Processor -- start 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 1 -- start start 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient: init 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient(hunting): picked mon.noname-a con 0x7f36d009ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient(hunting): picked mon.noname-c con 0x7f36d009c870 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient(hunting): picked mon.noname-b con 0x7f36d00b4360 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d009dec0 con 0x7f36d009ac40 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d009da30 con 0x7f36d009c870 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00b39c0 con 0x7f36d00b4360 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36d5f82640 10 monclient(hunting): authenticate will time out at 774.052827s 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36ceffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f36d009c870 0x7f36d013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38588/0 (socket says 172.21.15.81:38588) 2024-03-29T11:31:54.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.789+0000 7f36ceffd640 1 -- 172.21.15.81:0/1184726898 learned_addr learned my addr 172.21.15.81:0/1184726898 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569307997 0 0) 0x7f36d009da30 con 0x7f36d009c870 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b8003520 con 0x7f36d009c870 2024-03-29T11:31:54.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385943104 0 0) 0x7f36d00b39c0 con 0x7f36d00b4360 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d009da30 con 0x7f36d00b4360 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749609587 0 0) 0x7f36c0001110 con 0x7f36d009ac40 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d00b39c0 con 0x7f36d009ac40 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287184872 0 0) 0x7f36b8003520 con 0x7f36d009c870 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 >> v1:172.21.15.71:6790/0 conn(0x7f36d00b4360 legacy=0x7f36d014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 >> v1:172.21.15.71:6789/0 conn(0x7f36d009ac40 legacy=0x7f36d00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 monclient: found mon.noname-c 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00b5160 con 0x7f36d009c870 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f36c4001690 con 0x7f36d009c870 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: dump: 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.802 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c4001870 con 0x7f36d009c870 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 1 -- 172.21.15.81:0/1184726898 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f36c4001c80 con 0x7f36d009c870 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:54.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 4 set_mon_vals no callback set 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36ce7fc640 10 monclient: dump: 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.804 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 10 monclient: shutdown 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.790+0000 7f36d5f82640 1 -- 172.21.15.81:0/1184726898 >> v1:172.21.15.81:6789/0 conn(0x7f36d009c870 legacy=0x7f36d013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 1 -- 172.21.15.81:0/1184726898 shutdown_connections 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 1 -- 172.21.15.81:0/1184726898 >> 172.21.15.81:0/1184726898 conn(0x7f36d0066b00 msgr2=0x7f36d0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 1 -- 172.21.15.81:0/1184726898 shutdown_connections 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 1 -- 172.21.15.81:0/1184726898 wait complete. 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 10 monclient: build_initial_monmap 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 10 monclient: monmap: 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:54.805 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 1 librados: starting msgr at 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.791+0000 7f36d5f82640 1 librados: starting objecter 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 Processor -- start 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 -- start start 2024-03-29T11:31:54.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 librados: setting wanted keys 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 librados: calling monclient init 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient: init 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient(hunting): picked mon.noname-b con 0x7f36d009ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient(hunting): picked mon.noname-c con 0x7f36d01db230 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:54.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient(hunting): picked mon.noname-a con 0x7f36d00b0910 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d01da690 con 0x7f36d00b0910 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00b1910 con 0x7f36d009ac40 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d00acd00 con 0x7f36d01db230 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36d5f82640 10 monclient(hunting): authenticate will time out at 774.055849s 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36cf7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f36d009ac40 0x7f36d01dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38594/0 (socket says 172.21.15.81:38594) 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36cf7fe640 1 -- 172.21.15.81:0/1976456008 learned_addr learned my addr 172.21.15.81:0/1976456008 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36b7fff640 10 client.?.objecter ms_handle_connect 0x7f36d009ac40 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36b7fff640 10 client.?.objecter ms_handle_connect 0x7f36d00b0910 2024-03-29T11:31:54.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36b7fff640 10 client.?.objecter ms_handle_connect 0x7f36d01db230 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.792+0000 7f36b7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698792895 0 0) 0x7f36d00b1910 con 0x7f36d009ac40 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc004eb0 con 0x7f36d009ac40 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753470273 0 0) 0x7f36d01da690 con 0x7f36d00b0910 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.809 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d00b1910 con 0x7f36d00b0910 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353173383 0 0) 0x7f36d00acd00 con 0x7f36d01db230 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d01da690 con 0x7f36d01db230 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1470801000 0 0) 0x7f36bc004eb0 con 0x7f36d009ac40 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d00acd00 con 0x7f36d009ac40 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f36c00017e0 con 0x7f36d009ac40 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): dump: 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:54.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446477173 0 0) 0x7f36d00b1910 con 0x7f36d00b0910 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36bc004eb0 con 0x7f36d00b0910 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 284201294 0 0) 0x7f36d01da690 con 0x7f36d01db230 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d00b1910 con 0x7f36d01db230 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f36c8001b50 con 0x7f36d00b0910 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): dump: 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.812 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f36c40019e0 con 0x7f36d01db230 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient(hunting): dump: 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.813 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4124456970 0 0) 0x7f36bc004eb0 con 0x7f36d00b0910 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 >> v1:172.21.15.71:6790/0 conn(0x7f36d01db230 legacy=0x7f36d00b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 >> v1:172.21.15.81:6789/0 conn(0x7f36d009ac40 legacy=0x7f36d01dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:54.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 monclient: found mon.a 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d00ace90 con 0x7f36d00b0910 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 5 monclient: authenticate success, global_id 4140 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 10 monclient: _renew_subs 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d00af5f0 con 0x7f36d00b0910 2024-03-29T11:31:54.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 10 monclient: _renew_subs 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d01e3470 con 0x7f36d00b0910 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c80020c0 con 0x7f36d00b0910 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36d5f82640 1 librados: init done 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:54.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f36c8003970 con 0x7f36d00b0910 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 monclient: dump: 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:54.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36cdffb640 4 set_mon_vals no callback set 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f36c8003e60 con 0x7f36d00b0910 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 378661741 0 0) 0x7f36c8004620 con 0x7f36d00b0910 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 10 client.4140.objecter ms_dispatch 0x7f36d0000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.793+0000 7f36b7fff640 20 client.4140.objecter dump_active .. 0 homeless 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.794+0000 7f36d5f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.794+0000 7f36d5f82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.794+0000 7f36d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.794+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d009b020 con 0x7f36d00b0910 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.796+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f36c8004980 con 0x7f36d00b0910 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.796+0000 7f36b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:54.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.796+0000 7f36b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:54.819 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.811+0000 7f1bfcff1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:31:54.835 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:54.839+0000 7f90cbc41040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:31:54.854 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:54.859+0000 7f1bfcff1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:31:54.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.869+0000 7f36d5f82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a", "id": 4}] 2024-03-29T11:31:54.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.869+0000 7f36d5f82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a", "id": 4}] 2024-03-29T11:31:54.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.869+0000 7f36d5f82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:54.864 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.869+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a", "id": 4} v 0) v1 -- 0x7f36d004f730 con 0x7f36d00b0910 2024-03-29T11:31:54.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.870+0000 7f36b7fff640 1 -- 172.21.15.81:0/1976456008 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2378032413 0 1697012583) 0x7f36c8004110 con 0x7f36d00b0910 2024-03-29T11:31:54.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.870+0000 7f36b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a", "id": 4}] 2024-03-29T11:31:54.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.870+0000 7f36b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:54.866 INFO:teuthology.orchestra.run.smithi081.stdout:4 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 10 librados: watch_flush enter 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 10 librados: watch_flush exit 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 20 client.4140.objecter shutdown clearing up homeless session... 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 10 client.4140.objecter successfully canceled tick 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 10 monclient: shutdown 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 >> v1:172.21.15.71:6789/0 conn(0x7f36d00b0910 legacy=0x7f36d01e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 shutdown_connections 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 >> 172.21.15.81:0/1976456008 conn(0x7f36d0066b00 msgr2=0x7f36d00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:54.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 shutdown_connections 2024-03-29T11:31:54.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 1 -- 172.21.15.81:0/1976456008 wait complete. 2024-03-29T11:31:54.869 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:54.871+0000 7f36d5f82640 1 librados: shutdown 2024-03-29T11:31:54.877 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new ac76de25-e09f-44d1-841c-a04fa0b0590f 5 2024-03-29T11:31:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.025+0000 7fade593d640 10 monclient: get_monmap_and_config 2024-03-29T11:31:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.025+0000 7fade593d640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.026+0000 7fade593d640 10 monclient: monmap: 2024-03-29T11:31:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.020 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:55.026937+0000 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:55.026937+0000 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:55.021 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.026+0000 7fade593d640 1 Processor -- start 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.026+0000 7fade593d640 1 -- start start 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.026+0000 7fade593d640 10 monclient: init 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient(hunting): picked mon.noname-b con 0x7fade009ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient(hunting): picked mon.noname-a con 0x7fade009c870 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient(hunting): picked mon.noname-c con 0x7fade00b4360 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade009dec0 con 0x7fade009c870 2024-03-29T11:31:55.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade009da30 con 0x7fade00b4360 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00b39c0 con 0x7fade009ac40 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient(hunting): authenticate will time out at 774.290298s 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7faddf7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fade00b4360 0x7fade014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38608/0 (socket says 172.21.15.81:38608) 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7faddf7fe640 1 -- 172.21.15.81:0/4147940161 learned_addr learned my addr 172.21.15.81:0/4147940161 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545348636 0 0) 0x7fade009da30 con 0x7fade00b4360 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc8003520 con 0x7fade00b4360 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766279378 0 0) 0x7fadd0001110 con 0x7fade009c870 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade009da30 con 0x7fade009c870 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291164728 0 0) 0x7fade00b39c0 con 0x7fade009ac40 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd0001110 con 0x7fade009ac40 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1747478036 0 0) 0x7fadc8003520 con 0x7fade00b4360 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 >> v1:172.21.15.71:6790/0 conn(0x7fade009ac40 legacy=0x7fade00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 >> v1:172.21.15.71:6789/0 conn(0x7fade009c870 legacy=0x7fade013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 monclient: found mon.noname-c 2024-03-29T11:31:55.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00b5160 con 0x7fade00b4360 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fadd8001a10 con 0x7fade00b4360 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: dump: 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd8001c60 con 0x7fade00b4360 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fade593d640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 4 set_mon_vals no callback set 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.027+0000 7fadddffb640 1 -- 172.21.15.81:0/4147940161 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fadd8002030 con 0x7fade00b4360 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fadddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fadddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fadddffb640 10 monclient: dump: 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 10 monclient: shutdown 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 1 -- 172.21.15.81:0/4147940161 >> v1:172.21.15.81:6789/0 conn(0x7fade00b4360 legacy=0x7fade014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 1 -- 172.21.15.81:0/4147940161 shutdown_connections 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 1 -- 172.21.15.81:0/4147940161 >> 172.21.15.81:0/4147940161 conn(0x7fade0066b00 msgr2=0x7fade0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 1 -- 172.21.15.81:0/4147940161 shutdown_connections 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.028+0000 7fade593d640 1 -- 172.21.15.81:0/4147940161 wait complete. 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 10 monclient: monmap: 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 1 librados: starting msgr at 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 1 librados: starting objecter 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 1 Processor -- start 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 1 -- start start 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 1 librados: setting wanted keys 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 1 librados: calling monclient init 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.029+0000 7fade593d640 10 monclient: init 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient(hunting): picked mon.noname-a con 0x7fade009ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient(hunting): picked mon.noname-c con 0x7fade0146170 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient(hunting): picked mon.noname-b con 0x7fade01d6c00 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade0144000 con 0x7fade009ac40 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade0145700 con 0x7fade01d6c00 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade01459e0 con 0x7fade0146170 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fade593d640 10 monclient(hunting): authenticate will time out at 774.293409s 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7faddf7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fade01d6c00 0x7fade0136cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38616/0 (socket says 172.21.15.81:38616) 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7faddf7fe640 1 -- 172.21.15.81:0/1044674474 learned_addr learned my addr 172.21.15.81:0/1044674474 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 client.?.objecter ms_handle_connect 0x7fade01d6c00 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 client.?.objecter ms_handle_connect 0x7fade009ac40 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 client.?.objecter ms_handle_connect 0x7fade0146170 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787032574 0 0) 0x7fade0145700 con 0x7fade01d6c00 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc004eb0 con 0x7fade01d6c00 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592740273 0 0) 0x7fade0144000 con 0x7fade009ac40 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade0145700 con 0x7fade009ac40 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168633270 0 0) 0x7fade01459e0 con 0x7fade0146170 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fade0144000 con 0x7fade0146170 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2472926896 0 0) 0x7fadcc004eb0 con 0x7fade01d6c00 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade01459e0 con 0x7fade01d6c00 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fadd8001f70 con 0x7fade01d6c00 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): dump: 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.040 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 61246056 0 0) 0x7fade0145700 con 0x7fade009ac40 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadcc004eb0 con 0x7fade009ac40 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fadd4001870 con 0x7fade009ac40 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): dump: 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.041 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795281118 0 0) 0x7fade0144000 con 0x7fade0146170 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fade0145700 con 0x7fade0146170 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fadd0001a40 con 0x7fade0146170 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.030+0000 7fadc7fff640 10 monclient(hunting): dump: 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.042 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4220095630 0 0) 0x7fade01459e0 con 0x7fade01d6c00 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 >> v1:172.21.15.71:6790/0 conn(0x7fade0146170 legacy=0x7fade01438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 >> v1:172.21.15.71:6789/0 conn(0x7fade009ac40 legacy=0x7fade00b74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 monclient: found mon.b 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade0137400 con 0x7fade01d6c00 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 10 monclient: _renew_subs 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade0137650 con 0x7fade01d6c00 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 10 monclient: _renew_subs 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fade01e36b0 con 0x7fade01d6c00 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fade593d640 1 librados: init done 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadd8001550 con 0x7fade01d6c00 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fadd8004fb0 con 0x7fade01d6c00 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 monclient: dump: 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.045 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7faddd7fa640 4 set_mon_vals no callback set 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fadd8005520 con 0x7fade01d6c00 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3842401026 0 0) 0x7fadd8006cd0 con 0x7fade01d6c00 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 10 client.4141.objecter ms_dispatch 0x7fade0000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.031+0000 7fadc7fff640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.032+0000 7fade593d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.032+0000 7fade593d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.032+0000 7fade593d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.032+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadc0005080 con 0x7fade01d6c00 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.033+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fadd8006800 con 0x7fade01d6c00 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.033+0000 7fadc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.046 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.033+0000 7fadc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.106+0000 7fade593d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ac76de25-e09f-44d1-841c-a04fa0b0590f", "id": 5}] 2024-03-29T11:31:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.106+0000 7fade593d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ac76de25-e09f-44d1-841c-a04fa0b0590f", "id": 5}] 2024-03-29T11:31:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.106+0000 7fade593d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.106+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ac76de25-e09f-44d1-841c-a04fa0b0590f", "id": 5} v 0) v1 -- 0x7fadc00058b0 con 0x7fade01d6c00 2024-03-29T11:31:55.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.109+0000 7fadc7fff640 1 -- 172.21.15.81:0/1044674474 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ac76de25-e09f-44d1-841c-a04fa0b0590f", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3106158720 0 1988412176) 0x7fadd80069a0 con 0x7fade01d6c00 2024-03-29T11:31:55.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.109+0000 7fadc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ac76de25-e09f-44d1-841c-a04fa0b0590f", "id": 5}] 2024-03-29T11:31:55.104 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.109+0000 7fadc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:55.105 INFO:teuthology.orchestra.run.smithi081.stdout:5 2024-03-29T11:31:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 10 librados: watch_flush enter 2024-03-29T11:31:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 10 librados: watch_flush exit 2024-03-29T11:31:55.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 10 monclient: shutdown 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 >> v1:172.21.15.81:6789/0 conn(0x7fade01d6c00 legacy=0x7fade0136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 shutdown_connections 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.110+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 >> 172.21.15.81:0/1044674474 conn(0x7fade0066b00 msgr2=0x7fade013e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.111+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 shutdown_connections 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.111+0000 7fade593d640 1 -- 172.21.15.81:0/1044674474 wait complete. 2024-03-29T11:31:55.107 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.111+0000 7fade593d640 1 librados: shutdown 2024-03-29T11:31:55.117 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new 26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425 6 2024-03-29T11:31:55.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.223+0000 7fdceb006640 10 monclient: get_monmap_and_config 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.223+0000 7fdceb006640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.223+0000 7fdceb006640 10 monclient: monmap: 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:55.224211+0000 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:55.224211+0000 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.218 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:55.219 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:55.219 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.219 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.222+0000 7f90cbc41040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.223+0000 7fdceb006640 1 Processor -- start 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 1 -- start start 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient: init 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient(hunting): picked mon.noname-c con 0x7fdce409ac40 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient(hunting): picked mon.noname-b con 0x7fdce409c870 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient(hunting): picked mon.noname-a con 0x7fdce40b4360 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce409dec0 con 0x7fdce40b4360 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce409da30 con 0x7fdce409ac40 2024-03-29T11:31:55.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce40b39c0 con 0x7fdce409c870 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdceb006640 10 monclient(hunting): authenticate will time out at 774.487726s 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce8da2640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdce409ac40 0x7fdce40b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38632/0 (socket says 172.21.15.81:38632) 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce8da2640 1 -- 172.21.15.81:0/41318542 learned_addr learned my addr 172.21.15.81:0/41318542 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002161052 0 0) 0x7fdce409da30 con 0x7fdce409ac40 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce37fe640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.224+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd4003520 con 0x7fdce409ac40 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858921461 0 0) 0x7fdce40b39c0 con 0x7fdce409c870 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce409da30 con 0x7fdce409c870 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089891558 0 0) 0x7fdcdc001490 con 0x7fdce40b4360 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce40b39c0 con 0x7fdce40b4360 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1671590092 0 0) 0x7fdcd4003520 con 0x7fdce409ac40 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 >> v1:172.21.15.71:6790/0 conn(0x7fdce409c870 legacy=0x7fdce413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 >> v1:172.21.15.71:6789/0 conn(0x7fdce40b4360 legacy=0x7fdce414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 monclient: found mon.noname-c 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce40b5160 con 0x7fdce409ac40 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fdcd8001690 con 0x7fdce409ac40 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: dump: 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.233 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcd80017f0 con 0x7fdce409ac40 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 1 -- 172.21.15.81:0/41318542 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fdcd8001c00 con 0x7fdce409ac40 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 4 set_mon_vals no callback set 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdce37fe640 10 monclient: dump: 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.235 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 10 monclient: shutdown 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 1 -- 172.21.15.81:0/41318542 >> v1:172.21.15.81:6789/0 conn(0x7fdce409ac40 legacy=0x7fdce40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 1 -- 172.21.15.81:0/41318542 shutdown_connections 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 1 -- 172.21.15.81:0/41318542 >> 172.21.15.81:0/41318542 conn(0x7fdce4066b00 msgr2=0x7fdce4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 1 -- 172.21.15.81:0/41318542 shutdown_connections 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.225+0000 7fdceb006640 1 -- 172.21.15.81:0/41318542 wait complete. 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 10 monclient: monmap: 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:55.236 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 1 librados: starting msgr at 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 1 librados: starting objecter 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 1 Processor -- start 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 1 -- start start 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 1 librados: setting wanted keys 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 1 librados: calling monclient init 2024-03-29T11:31:55.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.226+0000 7fdceb006640 10 monclient: init 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient(hunting): picked mon.noname-c con 0x7fdce409ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient(hunting): picked mon.noname-b con 0x7fdce40acd70 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient(hunting): picked mon.noname-a con 0x7fdce40b06e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce41da800 con 0x7fdce40b06e0 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce411f830 con 0x7fdce40acd70 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdce41db3a0 con 0x7fdce409ac40 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdceb006640 10 monclient(hunting): authenticate will time out at 774.490809s 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdce40acd70 0x7fdce41dac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38634/0 (socket says 172.21.15.81:38634) 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce3fff640 1 -- 172.21.15.81:0/3044422775 learned_addr learned my addr 172.21.15.81:0/3044422775 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:55.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce17fa640 10 client.?.objecter ms_handle_connect 0x7fdce40acd70 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce17fa640 10 client.?.objecter ms_handle_connect 0x7fdce409ac40 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce17fa640 10 client.?.objecter ms_handle_connect 0x7fdce40b06e0 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.227+0000 7fdce17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007911538 0 0) 0x7fdce411f830 con 0x7fdce40acd70 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc8004eb0 con 0x7fdce40acd70 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021689644 0 0) 0x7fdce41db3a0 con 0x7fdce409ac40 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce411f830 con 0x7fdce409ac40 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342405479 0 0) 0x7fdce41da800 con 0x7fdce40b06e0 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce41db3a0 con 0x7fdce40b06e0 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1455845070 0 0) 0x7fdcc8004eb0 con 0x7fdce40acd70 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce41da800 con 0x7fdce40acd70 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fdcd0001a30 con 0x7fdce40acd70 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:55.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): dump: 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3690257755 0 0) 0x7fdce411f830 con 0x7fdce409ac40 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc8004eb0 con 0x7fdce409ac40 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fdcd8001b00 con 0x7fdce409ac40 2024-03-29T11:31:55.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): dump: 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2960021153 0 0) 0x7fdce41db3a0 con 0x7fdce40b06e0 2024-03-29T11:31:55.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdce411f830 con 0x7fdce40b06e0 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fdcdc001bc0 con 0x7fdce40b06e0 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient(hunting): dump: 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 528421985 0 0) 0x7fdce411f830 con 0x7fdce40b06e0 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 >> v1:172.21.15.71:6790/0 conn(0x7fdce409ac40 legacy=0x7fdce40af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 >> v1:172.21.15.81:6789/0 conn(0x7fdce40acd70 legacy=0x7fdce41dac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 monclient: found mon.a 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce41db530 con 0x7fdce40b06e0 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 5 monclient: authenticate success, global_id 4152 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 10 monclient: _renew_subs 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdce41e3300 con 0x7fdce40b06e0 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 10 client.4152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 10 monclient: _renew_subs 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdce41e3670 con 0x7fdce40b06e0 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.228+0000 7fdceb006640 1 librados: init done 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.229+0000 7fdceb006640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.229+0000 7fdceb006640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.229+0000 7fdceb006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.229+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdce409b020 con 0x7fdce40b06e0 2024-03-29T11:31:55.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcdc0014d0 con 0x7fdce40b06e0 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fdcdc001fd0 con 0x7fdce40b06e0 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 10 monclient: dump: 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.246 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce2ffd640 4 set_mon_vals no callback set 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdcdc003d20 con 0x7fdce40b06e0 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1662038036 0 0) 0x7fdcdc0054c0 con 0x7fdce40b06e0 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 10 client.4152.objecter ms_dispatch 0x7fdce4000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 3 client.4152.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 3 client.4152.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.230+0000 7fdce17fa640 20 client.4152.objecter dump_active .. 0 homeless 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.232+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdcdc004fe0 con 0x7fdce40b06e0 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.232+0000 7fdce17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.232+0000 7fdce17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:55.257 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.262+0000 7f1bfcff1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:31:55.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.305+0000 7fdceb006640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425", "id": 6}] 2024-03-29T11:31:55.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.305+0000 7fdceb006640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425", "id": 6}] 2024-03-29T11:31:55.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.305+0000 7fdceb006640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.305+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd new", "uuid": "26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425", "id": 6} v 0) v1 -- 0x7fdce404f730 con 0x7fdce40b06e0 2024-03-29T11:31:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdce17fa640 1 -- 172.21.15.81:0/3044422775 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2645041223 0 1113811849) 0x7fdcdc005270 con 0x7fdce40b06e0 2024-03-29T11:31:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdce17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425", "id": 6}] 2024-03-29T11:31:55.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdce17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:55.302 INFO:teuthology.orchestra.run.smithi081.stdout:6 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 10 librados: watch_flush enter 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 10 librados: watch_flush exit 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 20 client.4152.objecter shutdown clearing up homeless session... 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 10 client.4152.objecter successfully canceled tick 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 10 monclient: shutdown 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 >> v1:172.21.15.71:6789/0 conn(0x7fdce40b06e0 legacy=0x7fdce41e1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.307+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 shutdown_connections 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.308+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 >> 172.21.15.81:0/3044422775 conn(0x7fdce4066b00 msgr2=0x7fdce413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.308+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 shutdown_connections 2024-03-29T11:31:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.308+0000 7fdceb006640 1 -- 172.21.15.81:0/3044422775 wait complete. 2024-03-29T11:31:55.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.308+0000 7fdceb006640 1 librados: shutdown 2024-03-29T11:31:55.312 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd new eff27a86-368e-495c-8e23-1a1bcd717839 7 2024-03-29T11:31:55.354 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.359+0000 7f90cbc41040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:31:55.398 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.403+0000 7f1bfcff1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:31:55.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.417+0000 7f8548560640 10 monclient: get_monmap_and_config 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.417+0000 7f8548560640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.417+0000 7f8548560640 10 monclient: monmap: 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:55.418731+0000 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:55.418731+0000 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.418+0000 7f8548560640 1 Processor -- start 2024-03-29T11:31:55.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.418+0000 7f8548560640 1 -- start start 2024-03-29T11:31:55.414 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.418+0000 7f8548560640 10 monclient: init 2024-03-29T11:31:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient(hunting): picked mon.noname-a con 0x7f854009ac40 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient(hunting): picked mon.noname-c con 0x7f854009c870 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient(hunting): picked mon.noname-b con 0x7f85400b4360 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854009dec0 con 0x7f854009ac40 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854009da30 con 0x7f854009c870 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85400b39c0 con 0x7f85400b4360 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient(hunting): authenticate will time out at 774.682266s 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8545afb640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f854009c870 0x7f854013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38650/0 (socket says 172.21.15.81:38650) 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8545afb640 1 -- 172.21.15.81:0/2079304545 learned_addr learned my addr 172.21.15.81:0/2079304545 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235603239 0 0) 0x7f854009da30 con 0x7f854009c870 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852c003520 con 0x7f854009c870 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235034239 0 0) 0x7f8530001110 con 0x7f854009ac40 2024-03-29T11:31:55.432 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f854009da30 con 0x7f854009ac40 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611892213 0 0) 0x7f85400b39c0 con 0x7f85400b4360 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8530001110 con 0x7f85400b4360 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3326080191 0 0) 0x7f852c003520 con 0x7f854009c870 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 >> v1:172.21.15.71:6790/0 conn(0x7f85400b4360 legacy=0x7f854014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.433 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 >> v1:172.21.15.71:6789/0 conn(0x7f854009ac40 legacy=0x7f85400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 monclient: found mon.noname-c 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85400b5160 con 0x7f854009c870 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f853c001690 con 0x7f854009c870 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:55.434 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: dump: 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.435 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f853c001870 con 0x7f854009c870 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f8548560640 4 set_mon_vals no callback set 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 1 -- 172.21.15.81:0/2079304545 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f853c001c80 con 0x7f854009c870 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.419+0000 7f85452fa640 10 monclient: dump: 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.436 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.437 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 10 monclient: shutdown 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 1 -- 172.21.15.81:0/2079304545 >> v1:172.21.15.81:6789/0 conn(0x7f854009c870 legacy=0x7f854013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 1 -- 172.21.15.81:0/2079304545 shutdown_connections 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 1 -- 172.21.15.81:0/2079304545 >> 172.21.15.81:0/2079304545 conn(0x7f8540066b00 msgr2=0x7f8540066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 1 -- 172.21.15.81:0/2079304545 shutdown_connections 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.420+0000 7f8548560640 1 -- 172.21.15.81:0/2079304545 wait complete. 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 10 monclient: monmap: 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-29T11:31:55.438 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 1 librados: starting msgr at 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 1 librados: starting objecter 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 1 Processor -- start 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 1 -- start start 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 1 librados: setting wanted keys 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 1 librados: calling monclient init 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.421+0000 7f8548560640 10 monclient: init 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient(hunting): picked mon.noname-c con 0x7f854009ac40 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient(hunting): picked mon.noname-a con 0x7f85401d7bc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient(hunting): picked mon.noname-b con 0x7f85400b17f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854013b890 con 0x7f85401d7bc0 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85401d76a0 con 0x7f85400b17f0 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85401406b0 con 0x7f854009ac40 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8548560640 10 monclient(hunting): authenticate will time out at 774.685357s 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8546afd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f85400b17f0 0x7f854013b180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38664/0 (socket says 172.21.15.81:38664) 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8546afd640 1 -- 172.21.15.81:0/1811301053 learned_addr learned my addr 172.21.15.81:0/1811301053 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 client.?.objecter ms_handle_connect 0x7f85400b17f0 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 client.?.objecter ms_handle_connect 0x7f85401d7bc0 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 client.?.objecter ms_handle_connect 0x7f854009ac40 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062174853 0 0) 0x7f85401d76a0 con 0x7f85400b17f0 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8524004eb0 con 0x7f85400b17f0 2024-03-29T11:31:55.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345627 0 0) 0x7f854013b890 con 0x7f85401d7bc0 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85401d76a0 con 0x7f85401d7bc0 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329742169 0 0) 0x7f85401406b0 con 0x7f854009ac40 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f854013b890 con 0x7f854009ac40 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445678191 0 0) 0x7f8524004eb0 con 0x7f85400b17f0 2024-03-29T11:31:55.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85401406b0 con 0x7f85400b17f0 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8538001c70 con 0x7f85400b17f0 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3041071379 0 0) 0x7f85401d76a0 con 0x7f85401d7bc0 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8524004eb0 con 0x7f85401d7bc0 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f853c001b80 con 0x7f85401d7bc0 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.444 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714783786 0 0) 0x7f854013b890 con 0x7f854009ac40 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85401d76a0 con 0x7f854009ac40 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8530001860 con 0x7f854009ac40 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.422+0000 7f8536ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.445 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 729306096 0 0) 0x7f85401406b0 con 0x7f85400b17f0 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 >> v1:172.21.15.71:6790/0 conn(0x7f854009ac40 legacy=0x7f854013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 >> v1:172.21.15.71:6789/0 conn(0x7f85401d7bc0 legacy=0x7f854013aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 monclient: found mon.b 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8540140840 con 0x7f85400b17f0 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 10 monclient: _renew_subs 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85400b6f40 con 0x7f85400b17f0 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 10 monclient: _renew_subs 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85401d6f40 con 0x7f85400b17f0 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 1 librados: init done 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8538003990 con 0x7f85400b17f0 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8538003da0 con 0x7f85400b17f0 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8544af9640 4 set_mon_vals no callback set 2024-03-29T11:31:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 monclient: dump: 2024-03-29T11:31:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-29T11:31:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.448 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8538004350 con 0x7f85400b17f0 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1612029523 0 0) 0x7f8538006c80 con 0x7f85400b17f0 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 10 client.4153.objecter ms_dispatch 0x7f8540000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8536ffd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:31:55.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.423+0000 7f8548560640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.424+0000 7f8548560640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.424+0000 7f8548560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.424+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528005080 con 0x7f85400b17f0 2024-03-29T11:31:55.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.425+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f85380047e0 con 0x7f85400b17f0 2024-03-29T11:31:55.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.425+0000 7f8536ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.425+0000 7f8536ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:55.483 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.488+0000 7f90cbc41040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:31:55.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.498+0000 7f8548560640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eff27a86-368e-495c-8e23-1a1bcd717839", "id": 7}] 2024-03-29T11:31:55.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.498+0000 7f8548560640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eff27a86-368e-495c-8e23-1a1bcd717839", "id": 7}] 2024-03-29T11:31:55.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.498+0000 7f8548560640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.498+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd new", "uuid": "eff27a86-368e-495c-8e23-1a1bcd717839", "id": 7} v 0) v1 -- 0x7f85280058b0 con 0x7f85400b17f0 2024-03-29T11:31:55.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.502+0000 7f8536ffd640 1 -- 172.21.15.81:0/1811301053 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eff27a86-368e-495c-8e23-1a1bcd717839", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1439236611 0 1371665406) 0x7f8538006620 con 0x7f85400b17f0 2024-03-29T11:31:55.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.502+0000 7f8536ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eff27a86-368e-495c-8e23-1a1bcd717839", "id": 7}] 2024-03-29T11:31:55.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.502+0000 7f8536ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:55.498 INFO:teuthology.orchestra.run.smithi081.stdout:7 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 10 librados: watch_flush enter 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 10 librados: watch_flush exit 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 10 monclient: shutdown 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 >> v1:172.21.15.81:6789/0 conn(0x7f85400b17f0 legacy=0x7f854013b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 shutdown_connections 2024-03-29T11:31:55.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 >> 172.21.15.81:0/1811301053 conn(0x7f8540066b00 msgr2=0x7f854013d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 shutdown_connections 2024-03-29T11:31:55.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 1 -- 172.21.15.81:0/1811301053 wait complete. 2024-03-29T11:31:55.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-29T11:31:55.503+0000 7f8548560640 1 librados: shutdown 2024-03-29T11:31:55.508 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:31:55.508 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:31:55.511 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:31:55.511 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:31:55.511 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:31:55.513 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:31:55.513 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:31:55.513 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:31:55.515 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:31:55.515 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:31:55.515 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:31:55.517 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:31:55.517 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:31:55.518 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:31:55.519 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:31:55.520 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:31:55.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:31:55.522 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:31:55.522 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:31:55.522 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:31:55.524 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:31:55.525 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:31:55.525 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:31:55.529 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:31:55.530 DEBUG:teuthology.orchestra.run.smithi071:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:55.531 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.536+0000 7f1bfcff1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:31:55.538 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.542+0000 7f90cbc41040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:31:55.585 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.591+0000 7f1bfcff1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.652+0000 7f465e26e640 10 monclient: get_monmap_and_config 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.652+0000 7f465e26e640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.652+0000 7f465e26e640 10 monclient: monmap: 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.648 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:55.654026+0000 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:55.654026+0000 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.652+0000 7f465e26e640 1 Processor -- start 2024-03-29T11:31:55.649 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.652+0000 7f465e26e640 1 -- start start 2024-03-29T11:31:55.650 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.652+0000 7f465e26e640 10 monclient: init 2024-03-29T11:31:55.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:55.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient(hunting): picked mon.noname-a con 0x7f46580b8240 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient(hunting): picked mon.noname-b con 0x7f46580b1f60 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient(hunting): picked mon.noname-c con 0x7f465809c8c0 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.659 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465809bd80 con 0x7f46580b8240 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465809b960 con 0x7f465809c8c0 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46580b2ec0 con 0x7f46580b1f60 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f465e26e640 10 monclient(hunting): authenticate will time out at 769.808524s 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4657fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f46580b8240 0x7f465813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35704/0 (socket says 172.21.15.71:35704) 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4657fff640 1 -- 172.21.15.71:0/2490808189 learned_addr learned my addr 172.21.15.71:0/2490808189 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f46577fe640 1 --1- 172.21.15.71:0/2490808189 >> v1:172.21.15.71:6790/0 conn(0x7f46580b1f60 0x7f465809c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45418/0 (socket says 172.21.15.71:45418) 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409030971 0 0) 0x7f46580b2ec0 con 0x7f46580b1f60 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.660 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4640003520 con 0x7f46580b1f60 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814010603 0 0) 0x7f465809bd80 con 0x7f46580b8240 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46580b2ec0 con 0x7f46580b8240 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622483620 0 0) 0x7f465809b960 con 0x7f465809c8c0 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:31:55.661 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.653+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465809bd80 con 0x7f465809c8c0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287284113 0 0) 0x7f465809bd80 con 0x7f465809c8c0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 >> v1:172.21.15.71:6790/0 conn(0x7f46580b1f60 legacy=0x7f465809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 >> v1:172.21.15.71:6789/0 conn(0x7f46580b8240 legacy=0x7f465813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 monclient: found mon.noname-c 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465813fb60 con 0x7f465809c8c0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f46500018b0 con 0x7f465809c8c0 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.662 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: dump: 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f465e26e640 5 monclient: authenticate success, global_id 4180 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f465e26e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:55.663 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4650001a10 con 0x7f465809c8c0 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 1 -- 172.21.15.71:0/2490808189 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4650001e60 con 0x7f465809c8c0 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f4656ffd640 10 monclient: dump: 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.664 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f465e26e640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.654+0000 7f465e26e640 4 set_mon_vals no callback set 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 10 monclient: shutdown 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 1 -- 172.21.15.71:0/2490808189 >> v1:172.21.15.81:6789/0 conn(0x7f465809c8c0 legacy=0x7f465814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 1 -- 172.21.15.71:0/2490808189 shutdown_connections 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 1 -- 172.21.15.71:0/2490808189 >> 172.21.15.71:0/2490808189 conn(0x7f4658066b00 msgr2=0x7f4658066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 1 -- 172.21.15.71:0/2490808189 shutdown_connections 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.655+0000 7f465e26e640 1 -- 172.21.15.71:0/2490808189 wait complete. 2024-03-29T11:31:55.665 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 10 monclient: build_initial_monmap 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 10 monclient: monmap: 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 0 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 0.000000 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:created 0.000000 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:55.666 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 1 librados: starting msgr at 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 1 librados: starting objecter 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 1 Processor -- start 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 1 -- start start 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 1 librados: setting wanted keys 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 1 librados: calling monclient init 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.656+0000 7f465e26e640 10 monclient: init 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient(hunting): picked mon.noname-a con 0x7f465809c8c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:55.667 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient(hunting): picked mon.noname-b con 0x7f46580b8240 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f4657fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f465809c8c0 0x7f46581d7880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35710/0 (socket says 172.21.15.71:35710) 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient(hunting): picked mon.noname-c con 0x7f46581d7f90 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465813a000 con 0x7f465809c8c0 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46581d7360 con 0x7f46580b8240 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46581d3ea0 con 0x7f46581d7f90 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.657+0000 7f465e26e640 10 monclient(hunting): authenticate will time out at 769.813031s 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4657fff640 1 -- 172.21.15.71:0/184011570 learned_addr learned my addr 172.21.15.71:0/184011570 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:31:55.668 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 client.?.objecter ms_handle_connect 0x7f465809c8c0 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 client.?.objecter ms_handle_connect 0x7f46581d7f90 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 client.?.objecter ms_handle_connect 0x7f46580b8240 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786912427 0 0) 0x7f465813a000 con 0x7f465809c8c0 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4644004eb0 con 0x7f465809c8c0 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503118675 0 0) 0x7f46581d3ea0 con 0x7f46581d7f90 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f465813a000 con 0x7f46581d7f90 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430216447 0 0) 0x7f46581d7360 con 0x7f46580b8240 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:55.670 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46581d3ea0 con 0x7f46580b8240 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545544833 0 0) 0x7f46581d3ea0 con 0x7f46580b8240 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46581d7360 con 0x7f46580b8240 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784968930 0 0) 0x7f4644004eb0 con 0x7f465809c8c0 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46581d3ea0 con 0x7f465809c8c0 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3346424079 0 0) 0x7f465813a000 con 0x7f46581d7f90 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4644004eb0 con 0x7f46581d7f90 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f464c001a50 con 0x7f465809c8c0 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.675 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4648001950 con 0x7f46580b8240 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:55.676 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.677 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4650001d20 con 0x7f46581d7f90 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.678 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3809901370 0 0) 0x7f46581d3ea0 con 0x7f465809c8c0 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 >> v1:172.21.15.71:6790/0 conn(0x7f46581d7f90 legacy=0x7f46581d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 >> v1:172.21.15.81:6789/0 conn(0x7f46580b8240 legacy=0x7f46581382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 monclient: found mon.a 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46581d4030 con 0x7f465809c8c0 2024-03-29T11:31:55.679 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f4654ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 10 monclient: _renew_subs 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4658138f50 con 0x7f465809c8c0 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 10 monclient: _renew_subs 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.680 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4658138be0 con 0x7f465809c8c0 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.658+0000 7f465e26e640 1 librados: init done 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f464c001c50 con 0x7f465809c8c0 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f46567fc640 4 set_mon_vals no callback set 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f464c004c30 con 0x7f465809c8c0 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 10 monclient: dump: 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:epoch 1 2024-03-29T11:31:55.681 INFO:teuthology.orchestra.run.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f464c005120 con 0x7f465809c8c0 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f464c006a30 con 0x7f465809c8c0 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 10 client.4188.objecter ms_dispatch 0x7f4658000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:55.682 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.659+0000 7f4654ff9640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.660+0000 7f465e26e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.660+0000 7f465e26e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.660+0000 7f465e26e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.660+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4638005080 con 0x7f465809c8c0 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.662+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f464c006470 con 0x7f465809c8c0 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.662+0000 7f4654ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:55.683 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.662+0000 7f4654ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:55.684 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.689+0000 7f90cbc41040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:55.710 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.716+0000 7f1bfcff1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:55.718 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.722+0000 7f465e26e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:55.718 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.723+0000 7f465e26e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:55.718 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.723+0000 7f465e26e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:31:55.718 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.723+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4638005330 con 0x7f465809c8c0 2024-03-29T11:31:55.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.723+0000 7f4654ff9640 1 -- 172.21.15.71:0/184011570 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f464c0066c0 con 0x7f465809c8c0 2024-03-29T11:31:55.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.723+0000 7f4654ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:55.719 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.723+0000 7f4654ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:55.720 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-29T11:31:55.720 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:55.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 10 librados: watch_flush enter 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 10 librados: watch_flush exit 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 10 monclient: shutdown 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 >> v1:172.21.15.71:6789/0 conn(0x7f465809c8c0 legacy=0x7f46581d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 shutdown_connections 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 >> 172.21.15.71:0/184011570 conn(0x7f4658066b00 msgr2=0x7f4658069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:55.722 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 shutdown_connections 2024-03-29T11:31:55.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 1 -- 172.21.15.71:0/184011570 wait complete. 2024-03-29T11:31:55.723 INFO:teuthology.orchestra.run.smithi071.stderr:2024-03-29T11:31:55.724+0000 7f465e26e640 1 librados: shutdown 2024-03-29T11:31:55.730 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:31:55.730 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:31:55.783 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.788+0000 7f90cbc41040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:31:55.803 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.809+0000 7f1bfcff1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:31:55.940 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.945+0000 7f90cbc41040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:55.957 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:55.962+0000 7f1bfcff1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:31:55.982 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:55.987+0000 7f90cbc41040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:55.998 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:56.004+0000 7f1bfcff1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:31:56.121 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-03-29T11:31:56.124+0000 7f5626b14640 -1 Falling back to public interface 2024-03-29T11:31:56.123 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-03-29T11:31:56.126+0000 7f992d99a640 -1 Falling back to public interface 2024-03-29T11:31:56.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-03-29T11:31:56.126+0000 7f2192e15640 -1 Falling back to public interface 2024-03-29T11:31:56.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-03-29T11:31:56.126+0000 7f21917e1640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-29T11:31:56.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-03-29T11:31:56.126+0000 7f21917e1640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6802/0: (98) Address already in use 2024-03-29T11:31:56.128 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-03-29T11:31:56.132+0000 7f87a680c640 -1 Falling back to public interface 2024-03-29T11:31:56.134 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-03-29T11:31:56.139+0000 7ff5b82ac640 -1 Falling back to public interface 2024-03-29T11:31:56.135 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:56.140+0000 7f90cbc41040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:31:56.136 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-03-29T11:31:56.140+0000 7f50fecd3640 -1 Falling back to public interface 2024-03-29T11:31:56.141 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-03-29T11:31:56.144+0000 7f669a619640 -1 Falling back to public interface 2024-03-29T11:31:56.145 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-03-29T11:31:56.145+0000 7f2ebf711640 -1 Falling back to public interface 2024-03-29T11:31:56.155 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:56.158+0000 7f1bfcff1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:31:56.536 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:56.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.630+0000 7f45ce635640 10 monclient: get_monmap_and_config 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.630+0000 7f45ce635640 10 monclient: build_initial_monmap 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.630+0000 7f45ce635640 10 monclient: monmap: 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:56.632761+0000 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:56.632761+0000 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:56.627 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.628 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:56.628 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:31:56.628 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:31:56.628 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.631+0000 7f45ce635640 1 Processor -- start 2024-03-29T11:31:56.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.631+0000 7f45ce635640 1 -- start start 2024-03-29T11:31:56.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.631+0000 7f45ce635640 10 monclient: init 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient(hunting): picked mon.noname-b con 0x7f45c81422b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient(hunting): picked mon.noname-c con 0x7f45c8142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient(hunting): picked mon.noname-a con 0x7f45c80b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c8142c80 con 0x7f45c80b17d0 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c80aae40 con 0x7f45c8142690 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c813fbd0 con 0x7f45c81422b0 2024-03-29T11:31:56.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient(hunting): authenticate will time out at 770.787243s 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f45c81422b0 0x7f45c813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45434/0 (socket says 172.21.15.71:45434) 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c7fff640 1 -- 172.21.15.71:0/1113576381 learned_addr learned my addr 172.21.15.71:0/1113576381 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300817052 0 0) 0x7f45c813fbd0 con 0x7f45c81422b0 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b0003520 con 0x7f45c81422b0 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825881767 0 0) 0x7f45c8142c80 con 0x7f45c80b17d0 2024-03-29T11:31:56.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c813fbd0 con 0x7f45c80b17d0 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906548712 0 0) 0x7f45c80aae40 con 0x7f45c8142690 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:56.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c8142c80 con 0x7f45c8142690 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086544386 0 0) 0x7f45c813fbd0 con 0x7f45c80b17d0 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 >> v1:172.21.15.71:6790/0 conn(0x7f45c81422b0 legacy=0x7f45c813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 >> v1:172.21.15.81:6789/0 conn(0x7f45c8142690 legacy=0x7f45c80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8151940 con 0x7f45c80b17d0 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:56.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f45c00019f0 con 0x7f45c80b17d0 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: dump: 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.644 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c0001b50 con 0x7f45c80b17d0 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 1 -- 172.21.15.71:0/1113576381 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f45c0001f20 con 0x7f45c80b17d0 2024-03-29T11:31:56.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45c6ffd640 10 monclient: dump: 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:56.646 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.632+0000 7f45ce635640 4 set_mon_vals no callback set 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 10 monclient: shutdown 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 1 -- 172.21.15.71:0/1113576381 >> v1:172.21.15.71:6789/0 conn(0x7f45c80b17d0 legacy=0x7f45c80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 1 -- 172.21.15.71:0/1113576381 shutdown_connections 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 1 -- 172.21.15.71:0/1113576381 >> 172.21.15.71:0/1113576381 conn(0x7f45c8066a00 msgr2=0x7f45c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 1 -- 172.21.15.71:0/1113576381 shutdown_connections 2024-03-29T11:31:56.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.633+0000 7f45ce635640 1 -- 172.21.15.71:0/1113576381 wait complete. 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 10 monclient: build_initial_monmap 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 10 monclient: monmap: 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:31:56.648 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 1 librados: starting msgr at 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 1 librados: starting objecter 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 1 Processor -- start 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 1 -- start start 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 1 librados: setting wanted keys 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 1 librados: calling monclient init 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.634+0000 7f45ce635640 10 monclient: init 2024-03-29T11:31:56.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient(hunting): picked mon.noname-a con 0x7f45c80b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient(hunting): picked mon.noname-b con 0x7f45c81422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient(hunting): picked mon.noname-c con 0x7f45c8142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c80ad210 con 0x7f45c80b17d0 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c80ad480 con 0x7f45c81422b0 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c80ad760 con 0x7f45c8142690 2024-03-29T11:31:56.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient(hunting): authenticate will time out at 770.790224s 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c7fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f45c80b17d0 0x7f45c80ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35728/0 (socket says 172.21.15.71:35728) 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c7fff640 1 -- 172.21.15.71:0/1380989692 learned_addr learned my addr 172.21.15.71:0/1380989692 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 client.?.objecter ms_handle_connect 0x7f45c80b17d0 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 client.?.objecter ms_handle_connect 0x7f45c8142690 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553572355 0 0) 0x7f45c80ad210 con 0x7f45c80b17d0 2024-03-29T11:31:56.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45b4004eb0 con 0x7f45c80b17d0 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395225760 0 0) 0x7f45c80ad760 con 0x7f45c8142690 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c80ad210 con 0x7f45c8142690 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 client.?.objecter ms_handle_connect 0x7f45c81422b0 2024-03-29T11:31:56.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236464222 0 0) 0x7f45c80ad480 con 0x7f45c81422b0 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c80ad760 con 0x7f45c81422b0 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4028115956 0 0) 0x7f45b4004eb0 con 0x7f45c80b17d0 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45c80ad480 con 0x7f45c80b17d0 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 940536096 0 0) 0x7f45c80ad210 con 0x7f45c8142690 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45b4004eb0 con 0x7f45c8142690 2024-03-29T11:31:56.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f45b80013b0 con 0x7f45c80b17d0 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.654 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f45c00018b0 con 0x7f45c8142690 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.655 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117021006 0 0) 0x7f45c80ad760 con 0x7f45c81422b0 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45c80ad210 con 0x7f45c81422b0 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f45bc001810 con 0x7f45c81422b0 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:56.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:56.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:56.657 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:31:56.657 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.657 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.657 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.657 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 671454560 0 0) 0x7f45b4004eb0 con 0x7f45c8142690 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 >> v1:172.21.15.81:6789/0 conn(0x7f45c81422b0 legacy=0x7f45c812ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 >> v1:172.21.15.71:6789/0 conn(0x7f45c80b17d0 legacy=0x7f45c80ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 monclient: found mon.c 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c80abf60 con 0x7f45c8142690 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient: _renew_subs 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:31:56.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.635+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c81329a0 con 0x7f45c8142690 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 10 monclient: _renew_subs 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c812f510 con 0x7f45c8142690 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 1 librados: init done 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45c0003c20 con 0x7f45c8142690 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f45c0003f30 con 0x7f45c8142690 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c67fc640 4 set_mon_vals no callback set 2024-03-29T11:31:56.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 10 monclient: dump: 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:31:56.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f45c0004420 con 0x7f45c8142690 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f45c0005a10 con 0x7f45c8142690 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 10 client.4190.objecter ms_dispatch 0x7f45c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45c4ff9640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.636+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a8005080 con 0x7f45c8142690 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.638+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f45c0005340 con 0x7f45c8142690 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.638+0000 7f45c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:56.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.638+0000 7f45c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45ce635640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45ce635640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45ce635640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45a8005330 con 0x7f45c8142690 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45c4ff9640 1 -- 172.21.15.71:0/1380989692 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f45c00055c0 con 0x7f45c8142690 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.700+0000 7f45c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:56.697 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:31:56.697 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 10 librados: watch_flush enter 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 10 librados: watch_flush exit 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 10 monclient: shutdown 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 >> v1:172.21.15.71:6790/0 conn(0x7f45c8142690 legacy=0x7f45c8130140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 shutdown_connections 2024-03-29T11:31:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 >> 172.21.15.71:0/1380989692 conn(0x7f45c8066a00 msgr2=0x7f45c813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 shutdown_connections 2024-03-29T11:31:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 1 -- 172.21.15.71:0/1380989692 wait complete. 2024-03-29T11:31:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:31:56.701+0000 7f45ce635640 1 librados: shutdown 2024-03-29T11:31:56.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:56.962 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:56.967+0000 7f90cbc41040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:31:56.965 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:56.971+0000 7f1bfcff1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:31:57.236 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:57.240+0000 7f90cbc41040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:31:57.239 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:57.244+0000 7f1bfcff1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:31:57.288 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:57.291+0000 7f90cbc41040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:31:57.290 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:57.295+0000 7f1bfcff1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:31:57.400 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:57.405+0000 7f90cbc41040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:31:57.402 INFO:tasks.ceph.mgr.x.smithi081.stderr:2024-03-29T11:31:57.408+0000 7f1bfcff1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:31:58.414 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:58.418+0000 7f906c911640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:31:58.415 INFO:tasks.ceph.mgr.y.smithi071.stderr:2024-03-29T11:31:58.420+0000 7f906c911640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:31:59.582 INFO:tasks.ceph.osd.6.smithi081.stderr:2024-03-29T11:31:59.587+0000 7f992d99a640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:31:59.589 INFO:tasks.ceph.osd.3.smithi071.stderr:2024-03-29T11:31:59.593+0000 7f5626b14640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:31:59.633 INFO:tasks.ceph.osd.2.smithi071.stderr:2024-03-29T11:31:59.638+0000 7f87a680c640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:31:59.657 INFO:tasks.ceph.osd.4.smithi081.stderr:2024-03-29T11:31:59.662+0000 7ff5b82ac640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:31:59.657 INFO:tasks.ceph.osd.5.smithi081.stderr:2024-03-29T11:31:59.662+0000 7f2192e15640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:31:59.657 INFO:tasks.ceph.osd.7.smithi081.stderr:2024-03-29T11:31:59.662+0000 7f669a619640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:31:59.658 INFO:tasks.ceph.osd.1.smithi071.stderr:2024-03-29T11:31:59.661+0000 7f2ebf711640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:31:59.666 INFO:tasks.ceph.osd.0.smithi071.stderr:2024-03-29T11:31:59.671+0000 7f50fecd3640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:32:03.514 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:03.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.668+0000 7f696d525640 10 monclient: get_monmap_and_config 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.668+0000 7f696d525640 10 monclient: build_initial_monmap 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.668+0000 7f696d525640 10 monclient: monmap: 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:03.669761+0000 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:03.669761+0000 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:03.664 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.669+0000 7f696d525640 1 Processor -- start 2024-03-29T11:32:03.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.670+0000 7f696d525640 1 -- start start 2024-03-29T11:32:03.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.670+0000 7f696d525640 10 monclient: init 2024-03-29T11:32:03.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:03.683 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient(hunting): picked mon.noname-b con 0x7f69680b4970 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient(hunting): picked mon.noname-a con 0x7f69680b1e40 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient(hunting): picked mon.noname-c con 0x7f69680b3920 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69680aa930 con 0x7f69680b1e40 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69680aaba0 con 0x7f69680b3920 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69680aae80 con 0x7f69680b4970 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f696d525640 10 monclient(hunting): authenticate will time out at 777.825214s 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f6967fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f69680b3920 0x7f696814d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:53848/0 (socket says 172.21.15.71:53848) 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.671+0000 7f6967fff640 1 -- 172.21.15.71:0/3981506888 learned_addr learned my addr 172.21.15.71:0/3981506888 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203232628 0 0) 0x7f69680aaba0 con 0x7f69680b3920 2024-03-29T11:32:03.684 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6950003520 con 0x7f69680b3920 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216550669 0 0) 0x7f69680aa930 con 0x7f69680b1e40 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69680aaba0 con 0x7f69680b1e40 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833342770 0 0) 0x7f69680aae80 con 0x7f69680b4970 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:03.685 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69500039d0 con 0x7f69680b4970 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1732977900 0 0) 0x7f6950003520 con 0x7f69680b3920 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 >> v1:172.21.15.71:6790/0 conn(0x7f69680b4970 legacy=0x7f69680b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 >> v1:172.21.15.71:6789/0 conn(0x7f69680b1e40 legacy=0x7f69681453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 monclient: found mon.noname-c 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:32:03.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69680b3f00 con 0x7f69680b3920 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6958001690 con 0x7f69680b3920 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f6965ffb640 10 monclient: dump: 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:03.687 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f696d525640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.672+0000 7f696d525640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69580017f0 con 0x7f69680b3920 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f6965ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f6965ffb640 1 -- 172.21.15.71:0/3981506888 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6958001c00 con 0x7f69680b3920 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f6965ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f6965ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f6965ffb640 10 monclient: dump: 2024-03-29T11:32:03.688 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f696d525640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.673+0000 7f696d525640 4 set_mon_vals no callback set 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.674+0000 7f696d525640 10 monclient: shutdown 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.674+0000 7f696d525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.674+0000 7f696d525640 1 -- 172.21.15.71:0/3981506888 >> v1:172.21.15.81:6789/0 conn(0x7f69680b3920 legacy=0x7f696814d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.674+0000 7f696d525640 1 -- 172.21.15.71:0/3981506888 shutdown_connections 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.674+0000 7f696d525640 1 -- 172.21.15.71:0/3981506888 >> 172.21.15.71:0/3981506888 conn(0x7f6968066a00 msgr2=0x7f6968066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.675+0000 7f696d525640 1 -- 172.21.15.71:0/3981506888 shutdown_connections 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.675+0000 7f696d525640 1 -- 172.21.15.71:0/3981506888 wait complete. 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.676+0000 7f696d525640 10 monclient: build_initial_monmap 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.676+0000 7f696d525640 10 monclient: monmap: 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:03.690 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.676+0000 7f696d525640 1 librados: starting msgr at 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.676+0000 7f696d525640 1 librados: starting objecter 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.676+0000 7f696d525640 1 Processor -- start 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 1 -- start start 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 1 librados: setting wanted keys 2024-03-29T11:32:03.691 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 1 librados: calling monclient init 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 10 monclient: init 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.677+0000 7f696d525640 10 monclient(hunting): picked mon.noname-c con 0x7f69680b1e40 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 10 monclient(hunting): picked mon.noname-b con 0x7f69680b3920 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f6966ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f69680b1e40 0x7f6968139010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35876/0 (socket says 172.21.15.71:35876) 2024-03-29T11:32:03.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f6966ffd640 1 -- 172.21.15.71:0/4094312409 learned_addr learned my addr 172.21.15.71:0/4094312409 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 10 monclient(hunting): picked mon.noname-a con 0x7f69680b4970 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696813e1c0 con 0x7f69680b4970 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696813e430 con 0x7f69680b3920 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696813e710 con 0x7f69680b1e40 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f696d525640 10 monclient(hunting): authenticate will time out at 777.832284s 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f694f7fe640 10 client.?.objecter ms_handle_connect 0x7f69680b1e40 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f694f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f694f7fe640 10 client.?.objecter ms_handle_connect 0x7f69680b3920 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f694f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f694f7fe640 10 client.?.objecter ms_handle_connect 0x7f69680b4970 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.678+0000 7f694f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439154760 0 0) 0x7f696813e710 con 0x7f69680b1e40 2024-03-29T11:32:03.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6954004eb0 con 0x7f69680b1e40 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245459914 0 0) 0x7f696813e1c0 con 0x7f69680b4970 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696813e710 con 0x7f69680b4970 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193088056 0 0) 0x7f696813e430 con 0x7f69680b3920 2024-03-29T11:32:03.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.679+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f696813e1c0 con 0x7f69680b3920 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2017737313 0 0) 0x7f696813e1c0 con 0x7f69680b3920 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696813e430 con 0x7f69680b3920 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f695c0018b0 con 0x7f69680b3920 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:03.695 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1105425454 0 0) 0x7f696813e710 con 0x7f69680b4970 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696813e1c0 con 0x7f69680b4970 2024-03-29T11:32:03.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2266410813 0 0) 0x7f6954004eb0 con 0x7f69680b1e40 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f696813e710 con 0x7f69680b1e40 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6958001b00 con 0x7f69680b4970 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:03.697 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6960001cc0 con 0x7f69680b1e40 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.680+0000 7f694f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.698 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745646734 0 0) 0x7f696813e710 con 0x7f69680b1e40 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 >> v1:172.21.15.81:6789/0 conn(0x7f69680b3920 legacy=0x7f6968142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 >> v1:172.21.15.71:6789/0 conn(0x7f69680b4970 legacy=0x7f69681de6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:03.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 1 monclient: found mon.c 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696813ee80 con 0x7f69680b1e40 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 10 monclient: _renew_subs 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696813eb30 con 0x7f69680b1e40 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 10 monclient: _renew_subs 2024-03-29T11:32:03.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f696813b840 con 0x7f69680b1e40 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f696d525640 1 librados: init done 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69600039a0 con 0x7f69680b1e40 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f694f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.681+0000 7f69657fa640 4 set_mon_vals no callback set 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6960003d70 con 0x7f69680b1e40 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f694f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f694f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f696d525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f694f7fe640 10 monclient: dump: 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:03.701 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f696d525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.682+0000 7f696d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.683+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6948005080 con 0x7f69680b1e40 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.684+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f6960003f50 con 0x7f69680b1e40 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.685+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f6960084c20 con 0x7f69680b1e40 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.685+0000 7f694f7fe640 10 client.4235.objecter ms_dispatch 0x7f6968000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:03.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.685+0000 7f694f7fe640 3 client.4235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:03.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.686+0000 7f69667fc640 10 monclient: get_auth_request con 0x7f695406e1c0 auth_method 0 2024-03-29T11:32:03.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.686+0000 7f694f7fe640 3 client.4235.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:03.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.686+0000 7f694f7fe640 20 client.4235.objecter dump_active .. 0 homeless 2024-03-29T11:32:03.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.686+0000 7f694f7fe640 10 client.4235.objecter ms_handle_connect 0x7f695406e1c0 2024-03-29T11:32:03.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.692+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6960050560 con 0x7f69680b1e40 2024-03-29T11:32:03.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.692+0000 7f694f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:03.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:03.692+0000 7f694f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:04.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.074+0000 7f696d525640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:04.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.074+0000 7f696d525640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:04.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.074+0000 7f696d525640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:04.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.074+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6948005330 con 0x7f69680b1e40 2024-03-29T11:32:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.076+0000 7f694f7fe640 1 -- 172.21.15.71:0/4094312409 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f6960052490 con 0x7f69680b1e40 2024-03-29T11:32:04.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.076+0000 7f694f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:04.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.076+0000 7f694f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:04.074 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:04.074 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 10 librados: watch_flush enter 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 10 librados: watch_flush exit 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 20 client.4235.objecter shutdown clearing up homeless session... 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 10 client.4235.objecter successfully canceled tick 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 >> v1:172.21.15.71:6816/1596514235 conn(0x7f695406e1c0 legacy=0x7f6954070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 10 monclient: shutdown 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.080+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 >> v1:172.21.15.71:6790/0 conn(0x7f69680b1e40 legacy=0x7f6968139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.081+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 shutdown_connections 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.081+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 >> 172.21.15.71:0/4094312409 conn(0x7f6968066a00 msgr2=0x7f6968069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:04.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.081+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 shutdown_connections 2024-03-29T11:32:04.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.081+0000 7f696d525640 1 -- 172.21.15.71:0/4094312409 wait complete. 2024-03-29T11:32:04.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:04.082+0000 7f696d525640 1 librados: shutdown 2024-03-29T11:32:04.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:10.898 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.050+0000 7f69e3b6b640 10 monclient: get_monmap_and_config 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.050+0000 7f69e3b6b640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.050+0000 7f69e3b6b640 10 monclient: monmap: 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:11.051719+0000 2024-03-29T11:32:11.046 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:11.051719+0000 2024-03-29T11:32:11.047 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.047 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.047 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:11.047 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:11.047 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:11.047 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.051+0000 7f69e3b6b640 1 Processor -- start 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.052+0000 7f69e3b6b640 1 -- start start 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.052+0000 7f69e3b6b640 10 monclient: init 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.052+0000 7f69e3b6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc1422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc0b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc142c80 con 0x7f69dc1422b0 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0aae40 con 0x7f69dc142690 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc13fbd0 con 0x7f69dc0b17d0 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e3b6b640 10 monclient(hunting): authenticate will time out at 785.207153s 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e1106640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f69dc142690 0x7f69dc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:41116/0 (socket says 172.21.15.71:41116) 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.053+0000 7f69e1106640 1 -- 172.21.15.71:0/477101875 learned_addr learned my addr 172.21.15.71:0/477101875 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1841487345 0 0) 0x7f69dc142c80 con 0x7f69dc1422b0 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c8003520 con 0x7f69dc1422b0 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520200286 0 0) 0x7f69c8003520 con 0x7f69dc1422b0 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 >> v1:172.21.15.71:6790/0 conn(0x7f69dc0b17d0 legacy=0x7f69dc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 >> v1:172.21.15.81:6789/0 conn(0x7f69dc142690 legacy=0x7f69dc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 monclient: found mon.noname-a 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc151940 con 0x7f69dc1422b0 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f69cc001690 con 0x7f69dc1422b0 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e0905640 10 monclient: dump: 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.069 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e3b6b640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.054+0000 7f69e3b6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69cc001870 con 0x7f69dc1422b0 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e0905640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e3b6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e3b6b640 4 set_mon_vals no callback set 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e0905640 1 -- 172.21.15.71:0/477101875 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f69cc001c00 con 0x7f69dc1422b0 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e0905640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e0905640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.055+0000 7f69e0905640 10 monclient: dump: 2024-03-29T11:32:11.070 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 10 monclient: shutdown 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 1 -- 172.21.15.71:0/477101875 >> v1:172.21.15.71:6789/0 conn(0x7f69dc1422b0 legacy=0x7f69dc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 1 -- 172.21.15.71:0/477101875 shutdown_connections 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 1 -- 172.21.15.71:0/477101875 >> 172.21.15.71:0/477101875 conn(0x7f69dc066a00 msgr2=0x7f69dc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 1 -- 172.21.15.71:0/477101875 shutdown_connections 2024-03-29T11:32:11.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.056+0000 7f69e3b6b640 1 -- 172.21.15.71:0/477101875 wait complete. 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 10 monclient: build_initial_monmap 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 10 monclient: monmap: 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 1 librados: starting msgr at 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 1 librados: starting objecter 2024-03-29T11:32:11.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 1 Processor -- start 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 1 -- start start 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 1 librados: setting wanted keys 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 1 librados: calling monclient init 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.057+0000 7f69e3b6b640 10 monclient: init 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient(hunting): picked mon.noname-c con 0x7f69dc0b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient(hunting): picked mon.noname-b con 0x7f69dc1422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient(hunting): picked mon.noname-a con 0x7f69dc142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0b10e0 con 0x7f69dc142690 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0b1350 con 0x7f69dc1422b0 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc13c530 con 0x7f69dc0b17d0 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:11.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.058+0000 7f69e3b6b640 10 monclient(hunting): authenticate will time out at 785.212743s 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69e1106640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f69dc1422b0 0x7f69dc0af2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:41310/0 (socket says 172.21.15.71:41310) 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69e1106640 1 -- 172.21.15.71:0/1005062780 learned_addr learned my addr 172.21.15.71:0/1005062780 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 client.?.objecter ms_handle_connect 0x7f69dc142690 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 client.?.objecter ms_handle_connect 0x7f69dc1422b0 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645381352 0 0) 0x7f69dc0b10e0 con 0x7f69dc142690 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c0004eb0 con 0x7f69dc142690 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 client.?.objecter ms_handle_connect 0x7f69dc0b17d0 2024-03-29T11:32:11.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931913229 0 0) 0x7f69dc13c530 con 0x7f69dc0b17d0 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.059+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69dc0b10e0 con 0x7f69dc0b17d0 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227926427 0 0) 0x7f69c0004eb0 con 0x7f69dc142690 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69dc13c530 con 0x7f69dc142690 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1453766431 0 0) 0x7f69dc0b10e0 con 0x7f69dc0b17d0 2024-03-29T11:32:11.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69c0004eb0 con 0x7f69dc0b17d0 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f69cc001b80 con 0x7f69dc0b17d0 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): dump: 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:11.076 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f69d4001c80 con 0x7f69dc142690 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient(hunting): dump: 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:11.077 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3299528336 0 0) 0x7f69dc13c530 con 0x7f69dc142690 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 >> v1:172.21.15.71:6790/0 conn(0x7f69dc0b17d0 legacy=0x7f69dc13bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 >> v1:172.21.15.81:6789/0 conn(0x7f69dc1422b0 legacy=0x7f69dc0af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 monclient: found mon.a 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.060+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc14dc30 con 0x7f69dc142690 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:11.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 10 monclient: _renew_subs 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc14dee0 con 0x7f69dc142690 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:11.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 10 monclient: _renew_subs 2024-03-29T11:32:11.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:11.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc1e33b0 con 0x7f69dc142690 2024-03-29T11:32:11.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.061+0000 7f69e3b6b640 1 librados: init done 2024-03-29T11:32:11.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69d4003970 con 0x7f69dc142690 2024-03-29T11:32:11.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:11.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d3fff640 4 set_mon_vals no callback set 2024-03-29T11:32:11.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f69d4003c80 con 0x7f69dc142690 2024-03-29T11:32:11.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:11.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:11.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.062+0000 7f69d27fc640 10 monclient: dump: 2024-03-29T11:32:11.086 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:11.086 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:11.086 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.086 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:11.086 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:11.087 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:11.087 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:11.087 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:11.087 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:11.087 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:11.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.063+0000 7f69e3b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.064+0000 7f69e3b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.064+0000 7f69e3b6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:11.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.064+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c4005080 con 0x7f69dc142690 2024-03-29T11:32:11.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.064+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f69d4003de0 con 0x7f69dc142690 2024-03-29T11:32:11.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f69d4083c30 con 0x7f69dc142690 2024-03-29T11:32:11.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69d27fc640 10 client.4248.objecter ms_dispatch 0x7f69dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:11.090 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69d27fc640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:11.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69d27fc640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:11.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69e1907640 10 monclient: get_auth_request con 0x7f69c006de80 auth_method 0 2024-03-29T11:32:11.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69d27fc640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:32:11.091 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.065+0000 7f69d27fc640 10 client.4248.objecter ms_handle_connect 0x7f69c006de80 2024-03-29T11:32:11.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.076+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69d404f510 con 0x7f69dc142690 2024-03-29T11:32:11.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.077+0000 7f69d27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:11.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.077+0000 7f69d27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:11.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.386+0000 7f69e3b6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:11.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.387+0000 7f69e3b6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:11.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.387+0000 7f69e3b6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:11.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.387+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69c4005330 con 0x7f69dc142690 2024-03-29T11:32:11.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.387+0000 7f69d27fc640 1 -- 172.21.15.71:0/1005062780 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f69d4051440 con 0x7f69dc142690 2024-03-29T11:32:11.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.387+0000 7f69d27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:11.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.387+0000 7f69d27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:11.385 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:11.385 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:11.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 10 librados: watch_flush enter 2024-03-29T11:32:11.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 10 librados: watch_flush exit 2024-03-29T11:32:11.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:32:11.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:32:11.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 >> v1:172.21.15.71:6816/1596514235 conn(0x7f69c006de80 legacy=0x7f69c0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 10 monclient: shutdown 2024-03-29T11:32:11.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:11.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 >> v1:172.21.15.71:6789/0 conn(0x7f69dc142690 legacy=0x7f69dc0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:11.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 shutdown_connections 2024-03-29T11:32:11.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 >> 172.21.15.71:0/1005062780 conn(0x7f69dc066a00 msgr2=0x7f69dc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:11.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 shutdown_connections 2024-03-29T11:32:11.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 -- 172.21.15.71:0/1005062780 wait complete. 2024-03-29T11:32:11.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:11.391+0000 7f69e3b6b640 1 librados: shutdown 2024-03-29T11:32:11.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:18.209 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:18.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.359+0000 7ff62144a640 10 monclient: get_monmap_and_config 2024-03-29T11:32:18.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.359+0000 7ff62144a640 10 monclient: build_initial_monmap 2024-03-29T11:32:18.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.359+0000 7ff62144a640 10 monclient: monmap: 2024-03-29T11:32:18.355 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:18.360181+0000 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:18.360181+0000 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:18.356 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.360+0000 7ff62144a640 1 Processor -- start 2024-03-29T11:32:18.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.360+0000 7ff62144a640 1 -- start start 2024-03-29T11:32:18.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.360+0000 7ff62144a640 10 monclient: init 2024-03-29T11:32:18.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:18.359 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient(hunting): picked mon.noname-c con 0x7ff61c071de0 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient(hunting): picked mon.noname-b con 0x7ff61c0721c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient(hunting): picked mon.noname-a con 0x7ff61c1425d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c13fb10 con 0x7ff61c1425d0 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c142c20 con 0x7ff61c071de0 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0ada50 con 0x7ff61c0721c0 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff62144a640 10 monclient(hunting): authenticate will time out at 792.515761s 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.361+0000 7ff61affd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff61c071de0 0x7ff61c13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:44068/0 (socket says 172.21.15.71:44068) 2024-03-29T11:32:18.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.362+0000 7ff61affd640 1 -- 172.21.15.71:0/492707016 learned_addr learned my addr 172.21.15.71:0/492707016 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:18.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.363+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895670553 0 0) 0x7ff61c142c20 con 0x7ff61c071de0 2024-03-29T11:32:18.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.363+0000 7ff61a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:18.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.363+0000 7ff61a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:18.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.363+0000 7ff61a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:18.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.363+0000 7ff61a7fc640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:32:18.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.363+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff608003520 con 0x7ff61c071de0 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.365+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940755603 0 0) 0x7ff608003520 con 0x7ff61c071de0 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.365+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 >> v1:172.21.15.71:6790/0 conn(0x7ff61c0721c0 legacy=0x7ff61c145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.365+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 >> v1:172.21.15.71:6789/0 conn(0x7ff61c1425d0 legacy=0x7ff61c14da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:32:18.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0b0500 con 0x7ff61c071de0 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff60c001690 con 0x7ff61c071de0 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: dump: 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.371 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff62144a640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff62144a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff60c001920 con 0x7ff61c071de0 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff62144a640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff62144a640 4 set_mon_vals no callback set 2024-03-29T11:32:18.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 1 -- 172.21.15.71:0/492707016 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff60c001cb0 con 0x7ff61c071de0 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.367+0000 7ff61a7fc640 10 monclient: dump: 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 10 monclient: shutdown 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 1 -- 172.21.15.71:0/492707016 >> v1:172.21.15.81:6789/0 conn(0x7ff61c071de0 legacy=0x7ff61c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 1 -- 172.21.15.71:0/492707016 shutdown_connections 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 1 -- 172.21.15.71:0/492707016 >> 172.21.15.71:0/492707016 conn(0x7ff61c06b660 msgr2=0x7ff61c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 1 -- 172.21.15.71:0/492707016 shutdown_connections 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.368+0000 7ff62144a640 1 -- 172.21.15.71:0/492707016 wait complete. 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.369+0000 7ff62144a640 10 monclient: build_initial_monmap 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.369+0000 7ff62144a640 10 monclient: monmap: 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:18.374 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.370+0000 7ff62144a640 1 librados: starting msgr at 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.370+0000 7ff62144a640 1 librados: starting objecter 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.370+0000 7ff62144a640 1 Processor -- start 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.370+0000 7ff62144a640 1 -- start start 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.372+0000 7ff62144a640 1 librados: setting wanted keys 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.372+0000 7ff62144a640 1 librados: calling monclient init 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.372+0000 7ff62144a640 10 monclient: init 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.373+0000 7ff62144a640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.373+0000 7ff62144a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.374+0000 7ff62144a640 10 monclient(hunting): picked mon.noname-a con 0x7ff61c071de0 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:18.375 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.374+0000 7ff62144a640 10 monclient(hunting): picked mon.noname-b con 0x7ff61c1425d0 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:18.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff62144a640 10 monclient(hunting): picked mon.noname-c con 0x7ff61c141eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:18.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff62144a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c06f430 con 0x7ff61c071de0 2024-03-29T11:32:18.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff61affd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff61c071de0 0x7ff61c070720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45671/0 (socket says 172.21.15.71:45671) 2024-03-29T11:32:18.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff61affd640 1 -- 172.21.15.71:0/1508347383 learned_addr learned my addr 172.21.15.71:0/1508347383 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:18.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff6137fe640 10 client.?.objecter ms_handle_connect 0x7ff61c071de0 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff6137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c1dfe90 con 0x7ff61c1425d0 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c070e30 con 0x7ff61c141eb0 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff62144a640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.376+0000 7ff62144a640 10 monclient(hunting): authenticate will time out at 792.530455s 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 client.?.objecter ms_handle_connect 0x7ff61c1425d0 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793118510 0 0) 0x7ff61c06f430 con 0x7ff61c071de0 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5fc004eb0 con 0x7ff61c071de0 2024-03-29T11:32:18.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473926736 0 0) 0x7ff61c1dfe90 con 0x7ff61c1425d0 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61c06f430 con 0x7ff61c1425d0 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 client.?.objecter ms_handle_connect 0x7ff61c141eb0 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1167753090 0 0) 0x7ff5fc004eb0 con 0x7ff61c071de0 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff61c1dfe90 con 0x7ff61c071de0 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff60c001610 con 0x7ff61c071de0 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.377+0000 7ff6137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): dump: 2024-03-29T11:32:18.383 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866559089 0 0) 0x7ff61c06f430 con 0x7ff61c1425d0 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5fc004eb0 con 0x7ff61c1425d0 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301909846 0 0) 0x7ff61c070e30 con 0x7ff61c141eb0 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:18.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff61c06f430 con 0x7ff61c141eb0 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff6040017b0 con 0x7ff61c1425d0 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient(hunting): dump: 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:18.385 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3415819411 0 0) 0x7ff61c1dfe90 con 0x7ff61c071de0 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 >> v1:172.21.15.71:6790/0 conn(0x7ff61c141eb0 legacy=0x7ff61c1df780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 >> v1:172.21.15.81:6789/0 conn(0x7ff61c1425d0 legacy=0x7ff61c1df070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 monclient: found mon.a 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c06f6a0 con 0x7ff61c071de0 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:18.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff62144a640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff62144a640 10 monclient: _renew_subs 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff62144a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c06f980 con 0x7ff61c071de0 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff60c004bf0 con 0x7ff61c071de0 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff60c004f00 con 0x7ff61c071de0 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff6137fe640 10 monclient: dump: 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:18.387 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff62144a640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.378+0000 7ff62144a640 10 monclient: _renew_subs 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.379+0000 7ff62144a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.379+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c135f20 con 0x7ff61c071de0 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.379+0000 7ff62144a640 1 librados: init done 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.380+0000 7ff619ffb640 4 set_mon_vals no callback set 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.381+0000 7ff62144a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.381+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7ff60c01aed0 con 0x7ff61c071de0 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7ff60c084ba0 con 0x7ff61c071de0 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff6137fe640 10 client.4251.objecter ms_dispatch 0x7ff61c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:18.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff6137fe640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff6137fe640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff6137fe640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff62144a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff62144a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff600005080 con 0x7ff61c071de0 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff613fff640 10 monclient: get_auth_request con 0x7ff5fc06df70 auth_method 0 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.382+0000 7ff6137fe640 10 client.4251.objecter ms_handle_connect 0x7ff5fc06df70 2024-03-29T11:32:18.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.391+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff60c0504c0 con 0x7ff61c071de0 2024-03-29T11:32:18.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.391+0000 7ff6137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:18.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.391+0000 7ff6137fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:18.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.722+0000 7ff62144a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:18.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.722+0000 7ff62144a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:18.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.722+0000 7ff62144a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:18.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.722+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff600005330 con 0x7ff61c071de0 2024-03-29T11:32:18.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.723+0000 7ff6137fe640 1 -- 172.21.15.71:0/1508347383 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7ff60c0523f0 con 0x7ff61c071de0 2024-03-29T11:32:18.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.723+0000 7ff6137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:18.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.723+0000 7ff6137fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:18.719 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:18.719 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 10 librados: watch_flush enter 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 10 librados: watch_flush exit 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 >> v1:172.21.15.71:6816/1596514235 conn(0x7ff5fc06df70 legacy=0x7ff5fc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 10 monclient: shutdown 2024-03-29T11:32:18.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:18.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.727+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 >> v1:172.21.15.71:6789/0 conn(0x7ff61c071de0 legacy=0x7ff61c070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:18.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.728+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 shutdown_connections 2024-03-29T11:32:18.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.728+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 >> 172.21.15.71:0/1508347383 conn(0x7ff61c06b660 msgr2=0x7ff61c0a9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:18.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.728+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 shutdown_connections 2024-03-29T11:32:18.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.728+0000 7ff62144a640 1 -- 172.21.15.71:0/1508347383 wait complete. 2024-03-29T11:32:18.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:18.728+0000 7ff62144a640 1 librados: shutdown 2024-03-29T11:32:18.742 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:25.549 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.705+0000 7ff50163b640 10 monclient: get_monmap_and_config 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.705+0000 7ff50163b640 10 monclient: build_initial_monmap 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.705+0000 7ff50163b640 10 monclient: monmap: 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:25.706029+0000 2024-03-29T11:32:25.702 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:25.706029+0000 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.706+0000 7ff50163b640 1 Processor -- start 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.706+0000 7ff50163b640 1 -- start start 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.706+0000 7ff50163b640 10 monclient: init 2024-03-29T11:32:25.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4fc0b2800 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4fc0a9c00 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4fc0b38c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4fc141eb0 con 0x7ff4fc0b38c0 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4fc0b22d0 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4fc142d30 con 0x7ff4fc0b2800 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:25.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.707+0000 7ff50163b640 10 monclient(hunting): authenticate will time out at 799.861861s 2024-03-29T11:32:25.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.713+0000 7ff4fb7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff4fc0a9c00 0x7ff4fc13bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:58821/0 (socket says 172.21.15.71:58821) 2024-03-29T11:32:25.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.713+0000 7ff4fb7fe640 1 -- 172.21.15.71:0/928227105 learned_addr learned my addr 172.21.15.71:0/928227105 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:25.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051862517 0 0) 0x7ff4fc0b22d0 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e4003520 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482775113 0 0) 0x7ff4fc141eb0 con 0x7ff4fc0b38c0 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:32:25.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.714+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4fc0b22d0 con 0x7ff4fc0b38c0 2024-03-29T11:32:25.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.715+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2917622996 0 0) 0x7ff4e4003520 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.715+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 >> v1:172.21.15.71:6790/0 conn(0x7ff4fc0b2800 legacy=0x7ff4fc142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 >> v1:172.21.15.71:6789/0 conn(0x7ff4fc0b38c0 legacy=0x7ff4fc14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 1 monclient: found mon.noname-c 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc0aaf90 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:25.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff4f0001690 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.716+0000 7ff4faffd640 10 monclient: dump: 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:25.715 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.717+0000 7ff50163b640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.717+0000 7ff50163b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f00017f0 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff4faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff4faffd640 1 -- 172.21.15.71:0/928227105 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff4f0001b80 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff4faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff4faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff4faffd640 10 monclient: dump: 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:25.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff50163b640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.718+0000 7ff50163b640 4 set_mon_vals no callback set 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.719+0000 7ff50163b640 10 monclient: shutdown 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.719+0000 7ff50163b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.719+0000 7ff50163b640 1 -- 172.21.15.71:0/928227105 >> v1:172.21.15.81:6789/0 conn(0x7ff4fc0a9c00 legacy=0x7ff4fc13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.719+0000 7ff50163b640 1 -- 172.21.15.71:0/928227105 shutdown_connections 2024-03-29T11:32:25.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.719+0000 7ff50163b640 1 -- 172.21.15.71:0/928227105 >> 172.21.15.71:0/928227105 conn(0x7ff4fc066a00 msgr2=0x7ff4fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.720+0000 7ff50163b640 1 -- 172.21.15.71:0/928227105 shutdown_connections 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.720+0000 7ff50163b640 1 -- 172.21.15.71:0/928227105 wait complete. 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.721+0000 7ff50163b640 10 monclient: build_initial_monmap 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.721+0000 7ff50163b640 10 monclient: monmap: 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:25.720 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.721+0000 7ff50163b640 1 librados: starting msgr at 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.721+0000 7ff50163b640 1 librados: starting objecter 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.722+0000 7ff50163b640 1 Processor -- start 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.722+0000 7ff50163b640 1 -- start start 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.722+0000 7ff50163b640 1 librados: setting wanted keys 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.722+0000 7ff50163b640 1 librados: calling monclient init 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.722+0000 7ff50163b640 10 monclient: init 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4fc0a9c00 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4fc0b2800 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4fc0b38c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4fc06e3b0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4fc139eb0 con 0x7ff4fc0a9c00 2024-03-29T11:32:25.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4fc071ce0 con 0x7ff4fc0b38c0 2024-03-29T11:32:25.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:25.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.723+0000 7ff50163b640 10 monclient(hunting): authenticate will time out at 799.878036s 2024-03-29T11:32:25.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.727+0000 7ff4fb7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff4fc0b2800 0x7ff4fc139090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:58619/0 (socket says 172.21.15.71:58619) 2024-03-29T11:32:25.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.727+0000 7ff4fb7fe640 1 -- 172.21.15.71:0/354793143 learned_addr learned my addr 172.21.15.71:0/354793143 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:25.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.728+0000 7ff4f8ff9640 10 client.?.objecter ms_handle_connect 0x7ff4fc0b2800 2024-03-29T11:32:25.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.728+0000 7ff4f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.729+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22013453 0 0) 0x7ff4fc06e3b0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.729+0000 7ff4f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.729+0000 7ff4f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.729+0000 7ff4f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4e8004eb0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275282297 0 0) 0x7ff4e8004eb0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4fc06e3b0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff4f0001b00 con 0x7ff4fc0b2800 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:25.729 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.730+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1574450643 0 0) 0x7ff4fc06e3b0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 >> v1:172.21.15.71:6790/0 conn(0x7ff4fc0b38c0 legacy=0x7ff4fc1397a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 >> v1:172.21.15.81:6789/0 conn(0x7ff4fc0a9c00 legacy=0x7ff4fc06e840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 1 monclient: found mon.a 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc06ef50 con 0x7ff4fc0b2800 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:25.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff4f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff50163b640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff50163b640 10 monclient: _renew_subs 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff50163b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.731+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4fc06de90 con 0x7ff4fc0b2800 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff50163b640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff50163b640 10 monclient: _renew_subs 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff50163b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4fc13a310 con 0x7ff4fc0b2800 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff50163b640 1 librados: init done 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4f0004a90 con 0x7ff4fc0b2800 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff4f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.732+0000 7ff4f9ffb640 4 set_mon_vals no callback set 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff4f0004ea0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff4f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:25.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff4f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff4f8ff9640 10 monclient: dump: 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff50163b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff50163b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:25.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff50163b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:25.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.733+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4dc005080 con 0x7ff4fc0b2800 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.734+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7ff4f0005000 con 0x7ff4fc0b2800 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4fbfff640 10 monclient: get_auth_request con 0x7ff4e806dc40 auth_method 0 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4f8ff9640 10 client.4257.objecter ms_handle_connect 0x7ff4e806dc40 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7ff4f0084c10 con 0x7ff4fc0b2800 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4f8ff9640 10 client.4257.objecter ms_dispatch 0x7ff4fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4f8ff9640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4f8ff9640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:25.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.735+0000 7ff4f8ff9640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T11:32:25.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.742+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4f00504f0 con 0x7ff4fc0b2800 2024-03-29T11:32:25.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.742+0000 7ff4f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:25.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:25.743+0000 7ff4f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:26.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.077+0000 7ff50163b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:26.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.077+0000 7ff50163b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:26.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.077+0000 7ff50163b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:26.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.077+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4dc005330 con 0x7ff4fc0b2800 2024-03-29T11:32:26.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.078+0000 7ff4f8ff9640 1 -- 172.21.15.71:0/354793143 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7ff4f0052420 con 0x7ff4fc0b2800 2024-03-29T11:32:26.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.078+0000 7ff4f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:26.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.078+0000 7ff4f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:26.074 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:26.074 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:26.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.081+0000 7ff50163b640 10 librados: watch_flush enter 2024-03-29T11:32:26.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.082+0000 7ff50163b640 10 librados: watch_flush exit 2024-03-29T11:32:26.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.082+0000 7ff50163b640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T11:32:26.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.082+0000 7ff50163b640 10 client.4257.objecter successfully canceled tick 2024-03-29T11:32:26.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.084+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 >> v1:172.21.15.71:6816/1596514235 conn(0x7ff4e806dc40 legacy=0x7ff4e8070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.084+0000 7ff50163b640 10 monclient: shutdown 2024-03-29T11:32:26.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.085+0000 7ff50163b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:26.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.085+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 >> v1:172.21.15.71:6789/0 conn(0x7ff4fc0b2800 legacy=0x7ff4fc139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.086+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 shutdown_connections 2024-03-29T11:32:26.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.086+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 >> 172.21.15.71:0/354793143 conn(0x7ff4fc066a00 msgr2=0x7ff4fc1405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:26.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.087+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 shutdown_connections 2024-03-29T11:32:26.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.087+0000 7ff50163b640 1 -- 172.21.15.71:0/354793143 wait complete. 2024-03-29T11:32:26.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:26.088+0000 7ff50163b640 1 librados: shutdown 2024-03-29T11:32:26.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:32.910 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:33.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.476+0000 7f76c5ac3640 10 monclient: get_monmap_and_config 2024-03-29T11:32:33.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.476+0000 7f76c5ac3640 10 monclient: build_initial_monmap 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.476+0000 7f76c5ac3640 10 monclient: monmap: 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:33.477257+0000 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:33.477257+0000 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:33.472 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.473 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:33.473 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:33.473 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:33.473 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.477+0000 7f76c5ac3640 1 Processor -- start 2024-03-29T11:32:33.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.477+0000 7f76c5ac3640 1 -- start start 2024-03-29T11:32:33.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.477+0000 7f76c5ac3640 10 monclient: init 2024-03-29T11:32:33.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:33.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:33.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient(hunting): picked mon.noname-c con 0x7f76c01457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:33.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient(hunting): picked mon.noname-b con 0x7f76c013f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:33.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient(hunting): picked mon.noname-a con 0x7f76c00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c00aaef0 con 0x7f76c00a9d70 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c0145bd0 con 0x7f76c01457f0 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c0145eb0 con 0x7f76c013f650 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.478+0000 7f76c5ac3640 10 monclient(hunting): authenticate will time out at 807.632550s 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.481+0000 7f76bf7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f76c01457f0 0x7f76c013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:46877/0 (socket says 172.21.15.71:46877) 2024-03-29T11:32:33.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.481+0000 7f76bf7fe640 1 -- 172.21.15.71:0/1095438597 learned_addr learned my addr 172.21.15.71:0/1095438597 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 79713976 0 0) 0x7f76c0145eb0 con 0x7f76c013f650 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a8003520 con 0x7f76c013f650 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714120467 0 0) 0x7f76c00aaef0 con 0x7f76c00a9d70 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:33.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c0145eb0 con 0x7f76c00a9d70 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131191707 0 0) 0x7f76a8003520 con 0x7f76c013f650 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 >> v1:172.21.15.81:6789/0 conn(0x7f76c01457f0 legacy=0x7f76c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 >> v1:172.21.15.71:6789/0 conn(0x7f76c00a9d70 legacy=0x7f76c014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 monclient: found mon.noname-b 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c0146340 con 0x7f76c013f650 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:33.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f76b4001690 con 0x7f76c013f650 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76be7fc640 10 monclient: dump: 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:33.483 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:33.484 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:33.484 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76c5ac3640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:32:33.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.483+0000 7f76c5ac3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:33.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76b40017f0 con 0x7f76c013f650 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76be7fc640 1 -- 172.21.15.71:0/1095438597 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f76b4001c00 con 0x7f76c013f650 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76be7fc640 10 monclient: dump: 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:33.485 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76c5ac3640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.484+0000 7f76c5ac3640 4 set_mon_vals no callback set 2024-03-29T11:32:33.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 10 monclient: shutdown 2024-03-29T11:32:33.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:33.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1095438597 >> v1:172.21.15.71:6790/0 conn(0x7f76c013f650 legacy=0x7f76c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1095438597 shutdown_connections 2024-03-29T11:32:33.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1095438597 >> 172.21.15.71:0/1095438597 conn(0x7f76c0066a00 msgr2=0x7f76c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:33.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1095438597 shutdown_connections 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.485+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1095438597 wait complete. 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.486+0000 7f76c5ac3640 10 monclient: build_initial_monmap 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.486+0000 7f76c5ac3640 10 monclient: monmap: 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:33.489 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:33.490 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:33.490 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.490 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:33.490 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:33.490 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 1 librados: starting msgr at 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 1 librados: starting objecter 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 1 Processor -- start 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 1 -- start start 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 1 librados: setting wanted keys 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 1 librados: calling monclient init 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.487+0000 7f76c5ac3640 10 monclient: init 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:33.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient(hunting): picked mon.noname-a con 0x7f76c00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient(hunting): picked mon.noname-b con 0x7f76c013f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient(hunting): picked mon.noname-c con 0x7f76c01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c0141ec0 con 0x7f76c00a9d70 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c01421a0 con 0x7f76c013f650 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c0142a30 con 0x7f76c01457f0 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.488+0000 7f76c5ac3640 10 monclient(hunting): authenticate will time out at 807.642423s 2024-03-29T11:32:33.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76bffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f76c01457f0 0x7f76c01ddaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52805/0 (socket says 172.21.15.71:52805) 2024-03-29T11:32:33.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76bffff640 1 -- 172.21.15.71:0/1116368332 learned_addr learned my addr 172.21.15.71:0/1116368332 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:33.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 client.?.objecter ms_handle_connect 0x7f76c01457f0 2024-03-29T11:32:33.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:33.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 client.?.objecter ms_handle_connect 0x7f76c00a9d70 2024-03-29T11:32:33.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:33.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 client.?.objecter ms_handle_connect 0x7f76c013f650 2024-03-29T11:32:33.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952425398 0 0) 0x7f76c0141ec0 con 0x7f76c00a9d70 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.491+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ac004eb0 con 0x7f76c00a9d70 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511104905 0 0) 0x7f76c01421a0 con 0x7f76c013f650 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:33.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76c0141ec0 con 0x7f76c013f650 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.494+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998169138 0 0) 0x7f76ac004eb0 con 0x7f76c00a9d70 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76c01421a0 con 0x7f76c00a9d70 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f76b0001780 con 0x7f76c00a9d70 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): dump: 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:33.516 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278723499 0 0) 0x7f76c0142a30 con 0x7f76c01457f0 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:33.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.495+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ac004eb0 con 0x7f76c01457f0 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383315056 0 0) 0x7f76c0141ec0 con 0x7f76c013f650 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76c0142a30 con 0x7f76c013f650 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f76b4001b00 con 0x7f76c013f650 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 10 monclient(hunting): dump: 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:33.518 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3053301628 0 0) 0x7f76c01421a0 con 0x7f76c00a9d70 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 >> v1:172.21.15.71:6790/0 conn(0x7f76c01457f0 legacy=0x7f76c01ddaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.496+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 >> v1:172.21.15.81:6789/0 conn(0x7f76c013f650 legacy=0x7f76c01d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 1 monclient: found mon.a 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c0145170 con 0x7f76c00a9d70 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:33.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 10 monclient: _renew_subs 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c01425c0 con 0x7f76c00a9d70 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 10 monclient: _renew_subs 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c01d6e50 con 0x7f76c00a9d70 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76c5ac3640 1 librados: init done 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.497+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76b0001cb0 con 0x7f76c00a9d70 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.498+0000 7f76a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.498+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f76b0004980 con 0x7f76c00a9d70 2024-03-29T11:32:33.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.498+0000 7f76a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:33.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.498+0000 7f76a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:33.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.498+0000 7f76a7fff640 10 monclient: dump: 2024-03-29T11:32:33.521 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.498+0000 7f76c5ac3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.499+0000 7f76bdffb640 4 set_mon_vals no callback set 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.501+0000 7f76c5ac3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:33.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.501+0000 7f76c5ac3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.501+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c0142bc0 con 0x7f76c00a9d70 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.501+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f76b001a950 con 0x7f76c00a9d70 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.502+0000 7f76beffd640 10 monclient: get_auth_request con 0x7f76ac06df70 auth_method 0 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.502+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f76b0084880 con 0x7f76c00a9d70 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.502+0000 7f76a7fff640 10 client.4263.objecter ms_dispatch 0x7f76c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.502+0000 7f76a7fff640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.502+0000 7f76a7fff640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.502+0000 7f76a7fff640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.503+0000 7f76a7fff640 10 client.4263.objecter ms_handle_connect 0x7f76ac06df70 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.511+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76b0050160 con 0x7f76c00a9d70 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.511+0000 7f76a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:33.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.511+0000 7f76a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:33.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.835+0000 7f76c5ac3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:33.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.835+0000 7f76c5ac3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:33.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.835+0000 7f76c5ac3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:33.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.835+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76c0145300 con 0x7f76c00a9d70 2024-03-29T11:32:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.838+0000 7f76a7fff640 1 -- 172.21.15.71:0/1116368332 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f76b0052090 con 0x7f76c00a9d70 2024-03-29T11:32:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.838+0000 7f76a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:33.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.838+0000 7f76a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:33.833 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:33.834 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:33.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.841+0000 7f76c5ac3640 10 librados: watch_flush enter 2024-03-29T11:32:33.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.841+0000 7f76c5ac3640 10 librados: watch_flush exit 2024-03-29T11:32:33.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.841+0000 7f76c5ac3640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:32:33.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.841+0000 7f76c5ac3640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:32:33.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.842+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 >> v1:172.21.15.71:6816/1596514235 conn(0x7f76ac06df70 legacy=0x7f76ac070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.842+0000 7f76c5ac3640 10 monclient: shutdown 2024-03-29T11:32:33.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.842+0000 7f76c5ac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:33.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.842+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 >> v1:172.21.15.71:6789/0 conn(0x7f76c00a9d70 legacy=0x7f76c0137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:33.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.842+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 shutdown_connections 2024-03-29T11:32:33.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.842+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 >> 172.21.15.71:0/1116368332 conn(0x7f76c0066a00 msgr2=0x7f76c00b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:33.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.843+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 shutdown_connections 2024-03-29T11:32:33.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.843+0000 7f76c5ac3640 1 -- 172.21.15.71:0/1116368332 wait complete. 2024-03-29T11:32:33.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:33.843+0000 7f76c5ac3640 1 librados: shutdown 2024-03-29T11:32:33.857 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:40.664 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:40.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.821+0000 7f9ee4c40640 10 monclient: get_monmap_and_config 2024-03-29T11:32:40.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.821+0000 7f9ee4c40640 10 monclient: build_initial_monmap 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.821+0000 7f9ee4c40640 10 monclient: monmap: 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:40.822291+0000 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:40.822291+0000 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:40.817 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.823+0000 7f9ee4c40640 1 Processor -- start 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.824+0000 7f9ee4c40640 1 -- start start 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.824+0000 7f9ee4c40640 10 monclient: init 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.825+0000 7f9ee4c40640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.825+0000 7f9ee4c40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.825+0000 7f9ee4c40640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:40.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.825+0000 7f9ee4c40640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee013f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.827+0000 7f9ee4c40640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee00a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.827+0000 7f9ee4c40640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee00aaef0 con 0x7f9ee013f650 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.827+0000 7f9ee4c40640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee0145bd0 con 0x7f9ee00a9d70 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.827+0000 7f9ee4c40640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee0145eb0 con 0x7f9ee01457f0 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.827+0000 7f9ee4c40640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.827+0000 7f9ee4c40640 10 monclient(hunting): authenticate will time out at 814.981255s 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.828+0000 7f9ede59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f9ee01457f0 0x7f9ee013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53355/0 (socket says 172.21.15.71:53355) 2024-03-29T11:32:40.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.828+0000 7f9ede59c640 1 -- 172.21.15.71:0/228851160 learned_addr learned my addr 172.21.15.71:0/228851160 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:40.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878964986 0 0) 0x7f9ee0145eb0 con 0x7f9ee01457f0 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec8003520 con 0x7f9ee01457f0 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312333555 0 0) 0x7f9ec8003520 con 0x7f9ee01457f0 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 >> v1:172.21.15.81:6789/0 conn(0x7f9ee00a9d70 legacy=0x7f9ee014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 >> v1:172.21.15.71:6789/0 conn(0x7f9ee013f650 legacy=0x7f9ee00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 monclient: found mon.noname-b 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:40.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0146340 con 0x7f9ee01457f0 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: _finish_auth 0 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9ed4001690 con 0x7f9ee01457f0 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9edd59a640 10 monclient: dump: 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:40.827 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9ee4c40640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.830+0000 7f9ee4c40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ed4001860 con 0x7f9ee01457f0 2024-03-29T11:32:40.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9edd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:40.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9edd59a640 1 -- 172.21.15.71:0/228851160 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9ed4001c70 con 0x7f9ee01457f0 2024-03-29T11:32:40.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9edd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:40.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9edd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:40.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9edd59a640 10 monclient: dump: 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9ee4c40640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:40.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9ee4c40640 4 set_mon_vals no callback set 2024-03-29T11:32:40.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9ee4c40640 10 monclient: shutdown 2024-03-29T11:32:40.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9ee4c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:40.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.832+0000 7f9ee4c40640 1 -- 172.21.15.71:0/228851160 >> v1:172.21.15.71:6790/0 conn(0x7f9ee01457f0 legacy=0x7f9ee013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.833+0000 7f9ee4c40640 1 -- 172.21.15.71:0/228851160 shutdown_connections 2024-03-29T11:32:40.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.833+0000 7f9ee4c40640 1 -- 172.21.15.71:0/228851160 >> 172.21.15.71:0/228851160 conn(0x7f9ee0066a00 msgr2=0x7f9ee0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.834+0000 7f9ee4c40640 1 -- 172.21.15.71:0/228851160 shutdown_connections 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.834+0000 7f9ee4c40640 1 -- 172.21.15.71:0/228851160 wait complete. 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 10 monclient: build_initial_monmap 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 10 monclient: monmap: 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:40.833 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:40.834 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:40.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 1 librados: starting msgr at 2024-03-29T11:32:40.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 1 librados: starting objecter 2024-03-29T11:32:40.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 1 Processor -- start 2024-03-29T11:32:40.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 1 -- start start 2024-03-29T11:32:40.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 1 librados: setting wanted keys 2024-03-29T11:32:40.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 1 librados: calling monclient init 2024-03-29T11:32:40.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.835+0000 7f9ee4c40640 10 monclient: init 2024-03-29T11:32:40.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.836+0000 7f9ee4c40640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:40.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.836+0000 7f9ee4c40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:40.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.836+0000 7f9ee4c40640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:40.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.836+0000 7f9ee4c40640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee013f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:40.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.839+0000 7f9ee4c40640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:40.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.839+0000 7f9ee4c40640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee0144ba0 con 0x7f9ee00a9d70 2024-03-29T11:32:40.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.839+0000 7f9ee4c40640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee01383c0 con 0x7f9ee013f650 2024-03-29T11:32:40.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.839+0000 7f9ee4c40640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee0143af0 con 0x7f9ee01457f0 2024-03-29T11:32:40.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.839+0000 7f9ee4c40640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:40.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.839+0000 7f9ee4c40640 10 monclient(hunting): authenticate will time out at 814.993882s 2024-03-29T11:32:40.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.842+0000 7f9eddd9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9ee013f650 0x7f9ee013c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:54553/0 (socket says 172.21.15.71:54553) 2024-03-29T11:32:40.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.842+0000 7f9eddd9b640 1 -- 172.21.15.71:0/4241916840 learned_addr learned my addr 172.21.15.71:0/4241916840 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.842+0000 7f9eceffd640 10 client.?.objecter ms_handle_connect 0x7f9ee01457f0 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.842+0000 7f9eceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.843+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315435641 0 0) 0x7f9ee0143af0 con 0x7f9ee01457f0 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.843+0000 7f9eceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.843+0000 7f9eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.843+0000 7f9eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.843+0000 7f9eceffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:32:40.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.843+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec0004eb0 con 0x7f9ee01457f0 2024-03-29T11:32:40.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 10 client.?.objecter ms_handle_connect 0x7f9ee013f650 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283047526 0 0) 0x7f9ec0004eb0 con 0x7f9ee01457f0 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ee0143af0 con 0x7f9ee01457f0 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9ed8001b70 con 0x7f9ee01457f0 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 10 monclient(hunting): dump: 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:40.842 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.845+0000 7f9eceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4220831108 0 0) 0x7f9ee01383c0 con 0x7f9ee013f650 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec0005680 con 0x7f9ee013f650 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669476762 0 0) 0x7f9ee0143af0 con 0x7f9ee01457f0 2024-03-29T11:32:40.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 >> v1:172.21.15.81:6789/0 conn(0x7f9ee013f650 legacy=0x7f9ee013c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 >> v1:172.21.15.71:6789/0 conn(0x7f9ee00a9d70 legacy=0x7f9ee0137cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 monclient: found mon.c 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0143c80 con 0x7f9ee01457f0 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:40.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9eceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 10 monclient: _renew_subs 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee0143180 con 0x7f9ee01457f0 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 10 monclient: _renew_subs 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee013d9d0 con 0x7f9ee01457f0 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.846+0000 7f9ee4c40640 1 librados: init done 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.847+0000 7f9ee4c40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.847+0000 7f9ee4c40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.847+0000 7f9ee4c40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:40.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.847+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee0075f10 con 0x7f9ee01457f0 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ed8002100 con 0x7f9ee01457f0 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9eceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9ed8001b70 con 0x7f9ee01457f0 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9eceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9eceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9eceffd640 10 monclient: dump: 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:40.847 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.849+0000 7f9edcd99640 4 set_mon_vals no callback set 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.850+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f9ed80198c0 con 0x7f9ee01457f0 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.851+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f9ed8083870 con 0x7f9ee01457f0 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.851+0000 7f9eceffd640 10 client.4256.objecter ms_dispatch 0x7f9ee0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.851+0000 7f9eceffd640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:40.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.851+0000 7f9eceffd640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:40.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.851+0000 7f9eceffd640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:32:40.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.851+0000 7f9ede59c640 10 monclient: get_auth_request con 0x7f9ec006e070 auth_method 0 2024-03-29T11:32:40.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.856+0000 7f9eceffd640 10 client.4256.objecter ms_handle_connect 0x7f9ec006e070 2024-03-29T11:32:40.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.858+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ed804f0d0 con 0x7f9ee01457f0 2024-03-29T11:32:40.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.859+0000 7f9eceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:40.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:40.859+0000 7f9eceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:41.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.179+0000 7f9ee4c40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:41.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.179+0000 7f9ee4c40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:41.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.179+0000 7f9ee4c40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:41.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.179+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ee00b3680 con 0x7f9ee01457f0 2024-03-29T11:32:41.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.180+0000 7f9eceffd640 1 -- 172.21.15.71:0/4241916840 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f9ed804f0d0 con 0x7f9ee01457f0 2024-03-29T11:32:41.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.180+0000 7f9eceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:41.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.180+0000 7f9eceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:41.176 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:41.176 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:41.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.183+0000 7f9ee4c40640 10 librados: watch_flush enter 2024-03-29T11:32:41.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 10 librados: watch_flush exit 2024-03-29T11:32:41.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:32:41.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:32:41.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 >> v1:172.21.15.71:6816/1596514235 conn(0x7f9ec006e070 legacy=0x7f9ec0070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:41.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 10 monclient: shutdown 2024-03-29T11:32:41.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:41.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 >> v1:172.21.15.71:6790/0 conn(0x7f9ee01457f0 legacy=0x7f9ee013d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:41.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 shutdown_connections 2024-03-29T11:32:41.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.184+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 >> 172.21.15.71:0/4241916840 conn(0x7f9ee0066a00 msgr2=0x7f9ee0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:41.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.186+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 shutdown_connections 2024-03-29T11:32:41.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.186+0000 7f9ee4c40640 1 -- 172.21.15.71:0/4241916840 wait complete. 2024-03-29T11:32:41.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:41.186+0000 7f9ee4c40640 1 librados: shutdown 2024-03-29T11:32:41.199 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:48.006 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:48.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.165+0000 7fc0bd8d9640 10 monclient: get_monmap_and_config 2024-03-29T11:32:48.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.165+0000 7fc0bd8d9640 10 monclient: build_initial_monmap 2024-03-29T11:32:48.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.165+0000 7fc0bd8d9640 10 monclient: monmap: 2024-03-29T11:32:48.161 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:48.161 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:48.166583+0000 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:48.166583+0000 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:48.162 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.167+0000 7fc0bd8d9640 1 Processor -- start 2024-03-29T11:32:48.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.167+0000 7fc0bd8d9640 1 -- start start 2024-03-29T11:32:48.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.167+0000 7fc0bd8d9640 10 monclient: init 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0b80b46b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b80aeae0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b813fa10 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80b06a0 con 0x7fc0b80b46b0 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80b4230 con 0x7fc0b813fa10 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80b0910 con 0x7fc0b80aeae0 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:48.166 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.168+0000 7fc0bd8d9640 10 monclient(hunting): authenticate will time out at 822.322467s 2024-03-29T11:32:48.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.172+0000 7fc0b6ffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fc0b80b46b0 0x7fc0b80b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:33827/0 (socket says 172.21.15.71:33827) 2024-03-29T11:32:48.168 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.172+0000 7fc0b67fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fc0b80aeae0 0x7fc0b813f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45505/0 (socket says 172.21.15.71:45505) 2024-03-29T11:32:48.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.172+0000 7fc0b67fc640 1 -- 172.21.15.71:0/1692913763 learned_addr learned my addr 172.21.15.71:0/1692913763 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011456489 0 0) 0x7fc0b80b0910 con 0x7fc0b80aeae0 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a0003520 con 0x7fc0b80aeae0 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695469812 0 0) 0x7fc0b80b4230 con 0x7fc0b813fa10 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.175+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b80b0910 con 0x7fc0b813fa10 2024-03-29T11:32:48.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.176+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3762192511 0 0) 0x7fc0b80b0910 con 0x7fc0b813fa10 2024-03-29T11:32:48.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.176+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 >> v1:172.21.15.71:6790/0 conn(0x7fc0b80aeae0 legacy=0x7fc0b813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.176+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 >> v1:172.21.15.71:6789/0 conn(0x7fc0b80b46b0 legacy=0x7fc0b80b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.176+0000 7fc0b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.176+0000 7fc0b5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.176+0000 7fc0b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b80af6d0 con 0x7fc0b813fa10 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0bd8d9640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:32:48.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0bd8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc0ac001520 con 0x7fc0b813fa10 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.177+0000 7fc0b5ffb640 10 monclient: dump: 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:48.176 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0bd8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0ac001740 con 0x7fc0b813fa10 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0bd8d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0bd8d9640 4 set_mon_vals no callback set 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0b5ffb640 1 -- 172.21.15.71:0/1692913763 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc0ac001ad0 con 0x7fc0b813fa10 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.178+0000 7fc0b5ffb640 10 monclient: dump: 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:48.177 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.179+0000 7fc0bd8d9640 10 monclient: shutdown 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.179+0000 7fc0bd8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:48.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.179+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/1692913763 >> v1:172.21.15.81:6789/0 conn(0x7fc0b813fa10 legacy=0x7fc0b814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.184+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/1692913763 shutdown_connections 2024-03-29T11:32:48.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.184+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/1692913763 >> 172.21.15.71:0/1692913763 conn(0x7fc0b8066a00 msgr2=0x7fc0b8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:48.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.184+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/1692913763 shutdown_connections 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.184+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/1692913763 wait complete. 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.185+0000 7fc0bd8d9640 10 monclient: build_initial_monmap 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.185+0000 7fc0bd8d9640 10 monclient: monmap: 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:48.181 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.185+0000 7fc0bd8d9640 1 librados: starting msgr at 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.185+0000 7fc0bd8d9640 1 librados: starting objecter 2024-03-29T11:32:48.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.186+0000 7fc0bd8d9640 1 Processor -- start 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.186+0000 7fc0bd8d9640 1 -- start start 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.186+0000 7fc0bd8d9640 1 librados: setting wanted keys 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.186+0000 7fc0bd8d9640 1 librados: calling monclient init 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.186+0000 7fc0bd8d9640 10 monclient: init 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.187+0000 7fc0bd8d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.187+0000 7fc0bd8d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:48.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.187+0000 7fc0bd8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fc0b80aeae0 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fc0b80b46b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fc0b813fa10 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b8142720 con 0x7fc0b80aeae0 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b81e18b0 con 0x7fc0b80b46b0 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b8141600 con 0x7fc0b813fa10 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:48.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.190+0000 7fc0bd8d9640 10 monclient(hunting): authenticate will time out at 822.344442s 2024-03-29T11:32:48.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.193+0000 7fc0b67fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc0b80b46b0 0x7fc0b81e0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:44013/0 (socket says 172.21.15.71:44013) 2024-03-29T11:32:48.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.193+0000 7fc0b67fc640 1 -- 172.21.15.71:0/3306208322 learned_addr learned my addr 172.21.15.71:0/3306208322 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 client.?.objecter ms_handle_connect 0x7fc0b813fa10 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 client.?.objecter ms_handle_connect 0x7fc0b80aeae0 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195837328 0 0) 0x7fc0b8142720 con 0x7fc0b80aeae0 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:48.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4004eb0 con 0x7fc0b80aeae0 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529551534 0 0) 0x7fc0b8141600 con 0x7fc0b813fa10 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b8142720 con 0x7fc0b813fa10 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 client.?.objecter ms_handle_connect 0x7fc0b80b46b0 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763981788 0 0) 0x7fc0b81e18b0 con 0x7fc0b80b46b0 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:48.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0b8141600 con 0x7fc0b80b46b0 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.196+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3950978537 0 0) 0x7fc0b8142720 con 0x7fc0b813fa10 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b81e18b0 con 0x7fc0b813fa10 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc0ac0012c0 con 0x7fc0b813fa10 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): dump: 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.199 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2320199832 0 0) 0x7fc0b8141600 con 0x7fc0b80b46b0 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b8142720 con 0x7fc0b80b46b0 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc0a8001930 con 0x7fc0b80b46b0 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): dump: 2024-03-29T11:32:48.200 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659760270 0 0) 0x7fc0a4004eb0 con 0x7fc0b80aeae0 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0b8141600 con 0x7fc0b80aeae0 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc0b0001bf0 con 0x7fc0b80aeae0 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:48.201 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient(hunting): dump: 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 104545686 0 0) 0x7fc0b81e18b0 con 0x7fc0b813fa10 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 >> v1:172.21.15.81:6789/0 conn(0x7fc0b80b46b0 legacy=0x7fc0b81e0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 >> v1:172.21.15.71:6789/0 conn(0x7fc0b80aeae0 legacy=0x7fc0b81e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 monclient: found mon.c 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b8141790 con 0x7fc0b813fa10 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient: _finish_auth 0 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc09ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 10 monclient: _renew_subs 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b813b850 con 0x7fc0b813fa10 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 10 monclient: _renew_subs 2024-03-29T11:32:48.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:48.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b8142030 con 0x7fc0b813fa10 2024-03-29T11:32:48.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.197+0000 7fc0bd8d9640 1 librados: init done 2024-03-29T11:32:48.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.199+0000 7fc09dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:48.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.199+0000 7fc09dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:48.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.199+0000 7fc09dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:48.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.199+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b8143dc0 con 0x7fc0b813fa10 2024-03-29T11:32:48.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.203+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc0ac0038a0 con 0x7fc0b813fa10 2024-03-29T11:32:48.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.204+0000 7fc09ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:48.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.204+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc0ac0015e0 con 0x7fc0b813fa10 2024-03-29T11:32:48.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.204+0000 7fc09ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:48.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.204+0000 7fc09ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.204+0000 7fc09ffff640 10 monclient: dump: 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.204+0000 7fc0b57fa640 4 set_mon_vals no callback set 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.205+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fc0ac019a20 con 0x7fc0b813fa10 2024-03-29T11:32:48.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.205+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fc0ac083740 con 0x7fc0b813fa10 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.205+0000 7fc09ffff640 10 client.4262.objecter ms_dispatch 0x7fc0b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.205+0000 7fc09ffff640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.205+0000 7fc09ffff640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.205+0000 7fc09ffff640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.208+0000 7fc0b6ffd640 10 monclient: get_auth_request con 0x7fc0a406e0e0 auth_method 0 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.209+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc0ac04f020 con 0x7fc0b813fa10 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.209+0000 7fc09ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.209+0000 7fc09ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:48.207 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.209+0000 7fc09ffff640 10 client.4262.objecter ms_handle_connect 0x7fc0a406e0e0 2024-03-29T11:32:48.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.539+0000 7fc09dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:48.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.539+0000 7fc09dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:48.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.539+0000 7fc09dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:32:48.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.539+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0b8141920 con 0x7fc0b813fa10 2024-03-29T11:32:48.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.540+0000 7fc09ffff640 1 -- 172.21.15.71:0/3306208322 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fc0b8141920 con 0x7fc0b813fa10 2024-03-29T11:32:48.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.540+0000 7fc09ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:48.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.540+0000 7fc09ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:48.536 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:48.536 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 10 librados: watch_flush enter 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 10 librados: watch_flush exit 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 >> v1:172.21.15.71:6816/1596514235 conn(0x7fc0a406e0e0 legacy=0x7fc0a40705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 10 monclient: shutdown 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:48.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 >> v1:172.21.15.71:6790/0 conn(0x7fc0b813fa10 legacy=0x7fc0b81e11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:48.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 shutdown_connections 2024-03-29T11:32:48.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.544+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 >> 172.21.15.71:0/3306208322 conn(0x7fc0b8066a00 msgr2=0x7fc0b813f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:48.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.545+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 shutdown_connections 2024-03-29T11:32:48.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.545+0000 7fc09dffb640 1 -- 172.21.15.71:0/3306208322 wait complete. 2024-03-29T11:32:48.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:48.547+0000 7fc09dffb640 1 librados: shutdown 2024-03-29T11:32:48.559 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:55.366 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.524+0000 7f1a2333d640 10 monclient: get_monmap_and_config 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.524+0000 7f1a2333d640 10 monclient: build_initial_monmap 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.524+0000 7f1a2333d640 10 monclient: monmap: 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:32:55.526119+0000 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:32:55.526119+0000 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:32:55.523 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:32:55.524 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.526+0000 7f1a2333d640 1 Processor -- start 2024-03-29T11:32:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.526+0000 7f1a2333d640 1 -- start start 2024-03-29T11:32:55.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.526+0000 7f1a2333d640 10 monclient: init 2024-03-29T11:32:55.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:55.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:55.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1c0a9930 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:55.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a1c149dc0 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:55.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a1c14ff40 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:55.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0aae60 con 0x7f1a1c0a9930 2024-03-29T11:32:55.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0b4310 con 0x7f1a1c149dc0 2024-03-29T11:32:55.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c0afdd0 con 0x7f1a1c14ff40 2024-03-29T11:32:55.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:55.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.527+0000 7f1a2333d640 10 monclient(hunting): authenticate will time out at 829.682647s 2024-03-29T11:32:55.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.534+0000 7f1a210d9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1a1c0a9930 0x7f1a1c143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37531/0 (socket says 172.21.15.71:37531) 2024-03-29T11:32:55.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.534+0000 7f1a210d9640 1 -- 172.21.15.71:0/3964564258 learned_addr learned my addr 172.21.15.71:0/3964564258 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:55.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.535+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816598474 0 0) 0x7f1a1c0aae60 con 0x7f1a1c0a9930 2024-03-29T11:32:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.535+0000 7f1a13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.535+0000 7f1a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.535+0000 7f1a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.535+0000 7f1a13fff640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:32:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.535+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a08003520 con 0x7f1a1c0a9930 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259992896 0 0) 0x7f1a08003520 con 0x7f1a1c0a9930 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 >> v1:172.21.15.71:6790/0 conn(0x7f1a1c14ff40 legacy=0x7f1a1c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 >> v1:172.21.15.81:6789/0 conn(0x7f1a1c149dc0 legacy=0x7f1a1c14a240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 monclient: found mon.noname-a 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c14a950 con 0x7f1a1c0a9930 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1a0c001690 con 0x7f1a1c0a9930 2024-03-29T11:32:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: dump: 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a2333d640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a2333d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a0c001860 con 0x7f1a1c0a9930 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 1 -- 172.21.15.71:0/3964564258 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1a0c001c70 con 0x7f1a1c0a9930 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a13fff640 10 monclient: dump: 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.535 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a2333d640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.536+0000 7f1a2333d640 4 set_mon_vals no callback set 2024-03-29T11:32:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.537+0000 7f1a2333d640 10 monclient: shutdown 2024-03-29T11:32:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.537+0000 7f1a2333d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.537+0000 7f1a2333d640 1 -- 172.21.15.71:0/3964564258 >> v1:172.21.15.71:6789/0 conn(0x7f1a1c0a9930 legacy=0x7f1a1c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.540+0000 7f1a2333d640 1 -- 172.21.15.71:0/3964564258 shutdown_connections 2024-03-29T11:32:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.540+0000 7f1a2333d640 1 -- 172.21.15.71:0/3964564258 >> 172.21.15.71:0/3964564258 conn(0x7f1a1c0b35e0 msgr2=0x7f1a1c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:55.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.541+0000 7f1a2333d640 1 -- 172.21.15.71:0/3964564258 shutdown_connections 2024-03-29T11:32:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.543+0000 7f1a2333d640 1 -- 172.21.15.71:0/3964564258 wait complete. 2024-03-29T11:32:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.544+0000 7f1a2333d640 10 monclient: build_initial_monmap 2024-03-29T11:32:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.544+0000 7f1a2333d640 10 monclient: monmap: 2024-03-29T11:32:55.542 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.544+0000 7f1a2333d640 1 librados: starting msgr at 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.544+0000 7f1a2333d640 1 librados: starting objecter 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.544+0000 7f1a2333d640 1 Processor -- start 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 1 -- start start 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 1 librados: setting wanted keys 2024-03-29T11:32:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 1 librados: calling monclient init 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient: init 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1c0a9930 addr v1:172.21.15.71:6789/0 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient(hunting): picked mon.noname-c con 0x7f1a1c149dc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient(hunting): picked mon.noname-b con 0x7f1a1c14ff40 addr v1:172.21.15.81:6789/0 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c13c160 con 0x7f1a1c0a9930 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c13b780 con 0x7f1a1c14ff40 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a1c13ba60 con 0x7f1a1c149dc0 2024-03-29T11:32:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.545+0000 7f1a2333d640 10 monclient(hunting): authenticate will time out at 829.700907s 2024-03-29T11:32:55.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.549+0000 7f1a210d9640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f1a1c0a9930 0x7f1a1c138440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37707/0 (socket says 172.21.15.71:37707) 2024-03-29T11:32:55.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.549+0000 7f1a210d9640 1 -- 172.21.15.71:0/896030884 learned_addr learned my addr 172.21.15.71:0/896030884 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:32:55.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.549+0000 7f1a11ffb640 10 client.?.objecter ms_handle_connect 0x7f1a1c0a9930 2024-03-29T11:32:55.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.549+0000 7f1a11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 10 client.?.objecter ms_handle_connect 0x7f1a1c14ff40 2024-03-29T11:32:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171853613 0 0) 0x7f1a1c13c160 con 0x7f1a1c0a9930 2024-03-29T11:32:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.551+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a00004eb0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134105872 0 0) 0x7f1a1c13b780 con 0x7f1a1c14ff40 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a1c13c160 con 0x7f1a1c14ff40 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2996137044 0 0) 0x7f1a00004eb0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a1c13b780 con 0x7f1a1c0a9930 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1a0c001b70 con 0x7f1a1c0a9930 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): dump: 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:55.552 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 client.?.objecter ms_handle_connect 0x7f1a1c149dc0 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588003464 0 0) 0x7f1a1c13ba60 con 0x7f1a1c149dc0 2024-03-29T11:32:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a00004eb0 con 0x7f1a1c149dc0 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2487493158 0 0) 0x7f1a1c13b780 con 0x7f1a1c0a9930 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 >> v1:172.21.15.71:6790/0 conn(0x7f1a1c149dc0 legacy=0x7f1a1c13c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 >> v1:172.21.15.81:6789/0 conn(0x7f1a1c14ff40 legacy=0x7f1a1c2000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.554 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 monclient: found mon.a 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a1c13bd40 con 0x7f1a1c0a9930 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a2333d640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a2333d640 10 monclient: _renew_subs 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a2333d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.552+0000 7f1a2333d640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a1c13e430 con 0x7f1a1c0a9930 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.553+0000 7f1a2333d640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.553+0000 7f1a2333d640 10 monclient: _renew_subs 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.553+0000 7f1a2333d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:55.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.553+0000 7f1a2333d640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a1c13cef0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.553+0000 7f1a2333d640 1 librados: init done 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a0c004bf0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1a0c004f00 con 0x7f1a1c0a9930 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a11ffb640 10 monclient: dump: 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:32:55.556 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.554+0000 7f1a12ffd640 4 set_mon_vals no callback set 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.556+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f1a0c01aed0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f1a0c084d90 con 0x7f1a1c0a9930 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a11ffb640 10 client.4278.objecter ms_dispatch 0x7f1a1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a11ffb640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a11ffb640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a11ffb640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a077fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a077fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a1c075f10 con 0x7f1a1c0a9930 2024-03-29T11:32:55.557 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.557+0000 7f1a208d8640 10 monclient: get_auth_request con 0x7f1a0006ddf0 auth_method 0 2024-03-29T11:32:55.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.559+0000 7f1a11ffb640 10 client.4278.objecter ms_handle_connect 0x7f1a0006ddf0 2024-03-29T11:32:55.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.568+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a0c050670 con 0x7f1a1c0a9930 2024-03-29T11:32:55.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.568+0000 7f1a11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:55.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.568+0000 7f1a11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:55.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.889+0000 7f1a077fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:55.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.889+0000 7f1a077fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:55.885 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.889+0000 7f1a077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:32:55.886 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.889+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a1c0008d0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.891+0000 7f1a11ffb640 1 -- 172.21.15.71:0/896030884 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f1a0c0525a0 con 0x7f1a1c0a9930 2024-03-29T11:32:55.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.891+0000 7f1a11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:55.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.891+0000 7f1a11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:55.888 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:32:55.889 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 10 librados: watch_flush enter 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 10 librados: watch_flush exit 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 >> v1:172.21.15.71:6816/1596514235 conn(0x7f1a0006ddf0 legacy=0x7f1a000702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 10 monclient: shutdown 2024-03-29T11:32:55.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:55.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.895+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 >> v1:172.21.15.71:6789/0 conn(0x7f1a1c0a9930 legacy=0x7f1a1c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:55.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.896+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 shutdown_connections 2024-03-29T11:32:55.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.896+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 >> 172.21.15.71:0/896030884 conn(0x7f1a1c0b35e0 msgr2=0x7f1a1c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:55.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.896+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 shutdown_connections 2024-03-29T11:32:55.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.896+0000 7f1a077fe640 1 -- 172.21.15.71:0/896030884 wait complete. 2024-03-29T11:32:55.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:32:55.896+0000 7f1a077fe640 1 librados: shutdown 2024-03-29T11:32:55.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:02.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.891+0000 7f7e32d39640 10 monclient: get_monmap_and_config 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.891+0000 7f7e32d39640 10 monclient: build_initial_monmap 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.891+0000 7f7e32d39640 10 monclient: monmap: 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:02.892060+0000 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:02.892060+0000 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:02.887 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:02.888 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:02.888 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:02.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.892+0000 7f7e32d39640 1 Processor -- start 2024-03-29T11:33:02.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.892+0000 7f7e32d39640 1 -- start start 2024-03-29T11:33:02.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.892+0000 7f7e32d39640 10 monclient: init 2024-03-29T11:33:02.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:02.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:02.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c0b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c071eb0 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c143d10 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c06ef80 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c06f260 con 0x7f7e2c071eb0 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c0b40c0 con 0x7f7e2c143d10 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.893+0000 7f7e32d39640 10 monclient(hunting): authenticate will time out at 837.047660s 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.896+0000 7f7e30ad5640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7e2c0b1f60 0x7f7e2c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42225/0 (socket says 172.21.15.71:42225) 2024-03-29T11:33:02.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.896+0000 7f7e30ad5640 1 -- 172.21.15.71:0/721253739 learned_addr learned my addr 172.21.15.71:0/721253739 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:02.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.901+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394769381 0 0) 0x7f7e2c06ef80 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.901+0000 7f7e2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:02.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.901+0000 7f7e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:02.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.901+0000 7f7e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:02.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.901+0000 7f7e2b7fe640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:33:02.898 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.901+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e14003520 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.902+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119426726 0 0) 0x7f7e2c06f260 con 0x7f7e2c071eb0 2024-03-29T11:33:02.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.902+0000 7f7e2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:02.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.902+0000 7f7e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:02.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.902+0000 7f7e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:02.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.902+0000 7f7e2b7fe640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:33:02.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.902+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c06ef80 con 0x7f7e2c071eb0 2024-03-29T11:33:02.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281170140 0 0) 0x7f7e2c06ef80 con 0x7f7e2c071eb0 2024-03-29T11:33:02.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 >> v1:172.21.15.71:6790/0 conn(0x7f7e2c143d10 legacy=0x7f7e2c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:02.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 >> v1:172.21.15.71:6789/0 conn(0x7f7e2c0b1f60 legacy=0x7f7e2c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:02.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:02.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c155d70 con 0x7f7e2c071eb0 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.904+0000 7f7e2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e32d39640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e32d39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7e1c0015a0 con 0x7f7e2c071eb0 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e2b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e2b7fe640 10 monclient: dump: 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:02.901 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:02.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e32d39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e2b7fe640 1 -- 172.21.15.71:0/721253739 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e1c0017c0 con 0x7f7e2c071eb0 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e32d39640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.905+0000 7f7e32d39640 4 set_mon_vals no callback set 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.906+0000 7f7e32d39640 10 monclient: shutdown 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.906+0000 7f7e32d39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.906+0000 7f7e32d39640 1 -- 172.21.15.71:0/721253739 >> v1:172.21.15.81:6789/0 conn(0x7f7e2c071eb0 legacy=0x7f7e2c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.906+0000 7f7e32d39640 1 -- 172.21.15.71:0/721253739 shutdown_connections 2024-03-29T11:33:02.903 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.906+0000 7f7e32d39640 1 -- 172.21.15.71:0/721253739 >> 172.21.15.71:0/721253739 conn(0x7f7e2c066a00 msgr2=0x7f7e2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:02.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.910+0000 7f7e32d39640 1 -- 172.21.15.71:0/721253739 shutdown_connections 2024-03-29T11:33:02.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.910+0000 7f7e32d39640 1 -- 172.21.15.71:0/721253739 wait complete. 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.911+0000 7f7e32d39640 10 monclient: build_initial_monmap 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.911+0000 7f7e32d39640 10 monclient: monmap: 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:02.908 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:02.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.911+0000 7f7e32d39640 1 librados: starting msgr at 2024-03-29T11:33:02.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.911+0000 7f7e32d39640 1 librados: starting objecter 2024-03-29T11:33:02.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.912+0000 7f7e32d39640 1 Processor -- start 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.912+0000 7f7e32d39640 1 -- start start 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.912+0000 7f7e32d39640 1 librados: setting wanted keys 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.912+0000 7f7e32d39640 1 librados: calling monclient init 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.912+0000 7f7e32d39640 10 monclient: init 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient(hunting): picked mon.noname-a con 0x7f7e2c0b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient(hunting): picked mon.noname-c con 0x7f7e2c143d10 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient(hunting): picked mon.noname-b con 0x7f7e2c134670 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c1423c0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c134250 con 0x7f7e2c134670 2024-03-29T11:33:02.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c142f10 con 0x7f7e2c143d10 2024-03-29T11:33:02.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:02.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.913+0000 7f7e32d39640 10 monclient(hunting): authenticate will time out at 837.067356s 2024-03-29T11:33:02.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.917+0000 7f7e30ad5640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7e2c0b1f60 0x7f7e2c136890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42401/0 (socket says 172.21.15.71:42401) 2024-03-29T11:33:02.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.917+0000 7f7e30ad5640 1 -- 172.21.15.71:0/184387423 learned_addr learned my addr 172.21.15.71:0/184387423 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 client.?.objecter ms_handle_connect 0x7f7e2c0b1f60 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093041825 0 0) 0x7f7e2c1423c0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:02.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18004eb0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.920+0000 7f7e297fa640 10 client.?.objecter ms_handle_connect 0x7f7e2c143d10 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342973050 0 0) 0x7f7e18004eb0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e2c1423c0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107814064 0 0) 0x7f7e2c142f10 con 0x7f7e2c143d10 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18004eb0 con 0x7f7e2c143d10 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7e24001d10 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:02.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): dump: 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2807468605 0 0) 0x7f7e2c1423c0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 >> v1:172.21.15.71:6790/0 conn(0x7f7e2c143d10 legacy=0x7f7e2c132a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:02.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 >> v1:172.21.15.81:6789/0 conn(0x7f7e2c134670 legacy=0x7f7e2c1e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.921+0000 7f7e297fa640 1 monclient: found mon.a 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c141ea0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e297fa640 10 monclient: _finish_auth 0 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 10 monclient: _renew_subs 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c06f5d0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 10 monclient: _renew_subs 2024-03-29T11:33:02.923 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:02.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c1e75c0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.922+0000 7f7e32d39640 1 librados: init done 2024-03-29T11:33:02.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.923+0000 7f7e32d39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:02.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.923+0000 7f7e32d39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:02.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.923+0000 7f7e32d39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:02.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.923+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c075f10 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e24002010 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7e24004fb0 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e297fa640 10 monclient: dump: 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:02.925 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.927+0000 7f7e2a7fc640 4 set_mon_vals no callback set 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.928+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f7e24005110 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.929+0000 7f7e2bfff640 10 monclient: get_auth_request con 0x7f7e1806dd30 auth_method 0 2024-03-29T11:33:02.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.929+0000 7f7e297fa640 10 client.4284.objecter ms_handle_connect 0x7f7e1806dd30 2024-03-29T11:33:02.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.930+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f7e24084e50 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.930+0000 7f7e297fa640 10 client.4284.objecter ms_dispatch 0x7f7e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:02.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.930+0000 7f7e297fa640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:02.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.930+0000 7f7e297fa640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:02.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.930+0000 7f7e297fa640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T11:33:02.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.939+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e24050790 con 0x7f7e2c0b1f60 2024-03-29T11:33:02.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.939+0000 7f7e297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:02.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:02.939+0000 7f7e297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:03.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.276+0000 7f7e32d39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:03.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.276+0000 7f7e32d39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:03.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.276+0000 7f7e32d39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:03.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.276+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e2c142110 con 0x7f7e2c0b1f60 2024-03-29T11:33:03.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.279+0000 7f7e297fa640 1 -- 172.21.15.71:0/184387423 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f7e2c142110 con 0x7f7e2c0b1f60 2024-03-29T11:33:03.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.279+0000 7f7e297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:03.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.279+0000 7f7e297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:03.276 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:03.276 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:03.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.283+0000 7f7e32d39640 10 librados: watch_flush enter 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 10 librados: watch_flush exit 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 10 client.4284.objecter successfully canceled tick 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 >> v1:172.21.15.71:6816/1596514235 conn(0x7f7e1806dd30 legacy=0x7f7e180701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 10 monclient: shutdown 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:03.284 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.288+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 >> v1:172.21.15.71:6789/0 conn(0x7f7e2c0b1f60 legacy=0x7f7e2c136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:03.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.289+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 shutdown_connections 2024-03-29T11:33:03.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.289+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 >> 172.21.15.71:0/184387423 conn(0x7f7e2c066a00 msgr2=0x7f7e2c0b6930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:03.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.290+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 shutdown_connections 2024-03-29T11:33:03.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.290+0000 7f7e32d39640 1 -- 172.21.15.71:0/184387423 wait complete. 2024-03-29T11:33:03.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:03.290+0000 7f7e32d39640 1 librados: shutdown 2024-03-29T11:33:03.305 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:10.112 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:10.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.277+0000 7f82c0db5640 10 monclient: get_monmap_and_config 2024-03-29T11:33:10.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.277+0000 7f82c0db5640 10 monclient: build_initial_monmap 2024-03-29T11:33:10.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.277+0000 7f82c0db5640 10 monclient: monmap: 2024-03-29T11:33:10.273 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:10.273 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:10.278317+0000 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:10.278317+0000 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:10.274 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.278+0000 7f82c0db5640 1 Processor -- start 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.278+0000 7f82c0db5640 1 -- start start 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.278+0000 7f82c0db5640 10 monclient: init 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc0b0770 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient(hunting): picked mon.noname-c con 0x7f82bc14a700 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient(hunting): picked mon.noname-a con 0x7f82bc0ac5a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0aba70 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0aecb0 con 0x7f82bc14a700 2024-03-29T11:33:10.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0af150 con 0x7f82bc0b0770 2024-03-29T11:33:10.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:10.276 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82c0db5640 10 monclient(hunting): authenticate will time out at 844.433562s 2024-03-29T11:33:10.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82bad9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f82bc0ac5a0 0x7f82bc151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35186/0 (socket says 172.21.15.71:35186) 2024-03-29T11:33:10.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.279+0000 7f82bad9d640 1 -- 172.21.15.71:0/851591398 learned_addr learned my addr 172.21.15.71:0/851591398 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:10.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2124877634 0 0) 0x7f82bc0aba70 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:33:10.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a4003520 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578699403 0 0) 0x7f82bc0aecb0 con 0x7f82bc14a700 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc0aba70 con 0x7f82bc14a700 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805633203 0 0) 0x7f82bc0af150 con 0x7f82bc0b0770 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _negotiate 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:10.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc0aecb0 con 0x7f82bc0b0770 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2509796120 0 0) 0x7f82bc0aba70 con 0x7f82bc14a700 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 >> v1:172.21.15.71:6790/0 conn(0x7f82bc0b0770 legacy=0x7f82bc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 >> v1:172.21.15.71:6789/0 conn(0x7f82bc0ac5a0 legacy=0x7f82bc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: _finish_hunting 0 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 monclient: found mon.noname-c 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0ac9d0 con 0x7f82bc14a700 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: _finish_auth 0 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: _check_auth_tickets 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82c0db5640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:33:10.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82c0db5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f82b0001630 con 0x7f82bc14a700 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.280+0000 7f82b959a640 10 monclient: dump: 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b0001880 con 0x7f82bc14a700 2024-03-29T11:33:10.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82b959a640 1 -- 172.21.15.71:0/851591398 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f82b0001c10 con 0x7f82bc14a700 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82b959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82b959a640 10 monclient: dump: 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82c0db5640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:10.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.281+0000 7f82c0db5640 4 set_mon_vals no callback set 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.282+0000 7f82c0db5640 10 monclient: shutdown 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.282+0000 7f82c0db5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.282+0000 7f82c0db5640 1 -- 172.21.15.71:0/851591398 >> v1:172.21.15.81:6789/0 conn(0x7f82bc14a700 legacy=0x7f82bc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.282+0000 7f82c0db5640 1 -- 172.21.15.71:0/851591398 shutdown_connections 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.282+0000 7f82c0db5640 1 -- 172.21.15.71:0/851591398 >> 172.21.15.71:0/851591398 conn(0x7f82bc06b660 msgr2=0x7f82bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.283+0000 7f82c0db5640 1 -- 172.21.15.71:0/851591398 shutdown_connections 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.283+0000 7f82c0db5640 1 -- 172.21.15.71:0/851591398 wait complete. 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.284+0000 7f82c0db5640 10 monclient: build_initial_monmap 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.284+0000 7f82c0db5640 10 monclient: monmap: 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:10.285 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.284+0000 7f82c0db5640 1 librados: starting msgr at 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.284+0000 7f82c0db5640 1 librados: starting objecter 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 1 Processor -- start 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 1 -- start start 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 1 librados: setting wanted keys 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 1 librados: calling monclient init 2024-03-29T11:33:10.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 10 monclient: init 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.285+0000 7f82c0db5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 10 monclient(hunting): picked mon.noname-a con 0x7f82bc0ac5a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 10 monclient(hunting): picked mon.noname-c con 0x7f82bc0b0770 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc127260 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc1239d0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc1281f0 con 0x7f82bc127260 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc125e70 con 0x7f82bc0b0770 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82c0db5640 10 monclient(hunting): authenticate will time out at 844.440204s 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82bad9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f82bc127260 0x7f82bc1e70d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:36440/0 (socket says 172.21.15.71:36440) 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82bad9d640 1 -- 172.21.15.71:0/3556206042 learned_addr learned my addr 172.21.15.71:0/3556206042 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82aaffd640 10 client.?.objecter ms_handle_connect 0x7f82bc127260 2024-03-29T11:33:10.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82aaffd640 10 client.?.objecter ms_handle_connect 0x7f82bc0ac5a0 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.286+0000 7f82aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070613508 0 0) 0x7f82bc1281f0 con 0x7f82bc127260 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c004eb0 con 0x7f82bc127260 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 client.?.objecter ms_handle_connect 0x7f82bc0b0770 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585875220 0 0) 0x7f82bc1239d0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:10.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc1281f0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594388254 0 0) 0x7f82bc125e70 con 0x7f82bc0b0770 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc1239d0 con 0x7f82bc0b0770 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1979585450 0 0) 0x7f82bc1281f0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc125e70 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f82ac0017d0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): dump: 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:10.289 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.287+0000 7f82aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2362346999 0 0) 0x7f829c004eb0 con 0x7f82bc127260 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc1281f0 con 0x7f82bc127260 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f82b4001c00 con 0x7f82bc127260 2024-03-29T11:33:10.290 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient(hunting): dump: 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754839574 0 0) 0x7f82bc125e70 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.291 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 >> v1:172.21.15.71:6790/0 conn(0x7f82bc0b0770 legacy=0x7f82bc1dbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 >> v1:172.21.15.81:6789/0 conn(0x7f82bc127260 legacy=0x7f82bc1e70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 monclient: found mon.a 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc126dd0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 10 monclient: _renew_subs 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc126390 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:10.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 10 monclient: _renew_subs 2024-03-29T11:33:10.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:10.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc1454f0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.288+0000 7f82c0db5640 1 librados: init done 2024-03-29T11:33:10.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82ac001c60 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:10.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82b8d99640 4 set_mon_vals no callback set 2024-03-29T11:33:10.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f82ac0037b0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:10.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:10.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.289+0000 7f82aaffd640 10 monclient: dump: 2024-03-29T11:33:10.294 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:10.294 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.290+0000 7f82c0db5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.290+0000 7f82c0db5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.290+0000 7f82c0db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.290+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82a0005080 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.291+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f82ac003910 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f82ac083710 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82aaffd640 10 client.4290.objecter ms_dispatch 0x7f82bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:10.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82aaffd640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82aaffd640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82aaffd640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82b9d9b640 10 monclient: get_auth_request con 0x7f829c06dee0 auth_method 0 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.292+0000 7f82aaffd640 10 client.4290.objecter ms_handle_connect 0x7f829c06dee0 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.299+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82ac04eff0 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.299+0000 7f82aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:10.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.299+0000 7f82aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.665+0000 7f82c0db5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.665+0000 7f82c0db5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:10.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.665+0000 7f82c0db5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:33:10.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.665+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82a0005330 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.667+0000 7f82aaffd640 1 -- 172.21.15.71:0/3556206042 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f82ac050f20 con 0x7f82bc0ac5a0 2024-03-29T11:33:10.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.667+0000 7f82aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:10.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.667+0000 7f82aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:10.663 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:10.663 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:10.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 10 librados: watch_flush enter 2024-03-29T11:33:10.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 10 librados: watch_flush exit 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 >> v1:172.21.15.71:6816/1596514235 conn(0x7f829c06dee0 legacy=0x7f829c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 10 monclient: shutdown 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 >> v1:172.21.15.71:6789/0 conn(0x7f82bc0ac5a0 legacy=0x7f82bc144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 shutdown_connections 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 >> 172.21.15.71:0/3556206042 conn(0x7f82bc06b660 msgr2=0x7f82bc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 shutdown_connections 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 -- 172.21.15.71:0/3556206042 wait complete. 2024-03-29T11:33:10.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:10.671+0000 7f82c0db5640 1 librados: shutdown 2024-03-29T11:33:10.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:17.489 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:17.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.627+0000 7fa7c4b34640 10 monclient: get_monmap_and_config 2024-03-29T11:33:17.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.627+0000 7fa7c4b34640 10 monclient: build_initial_monmap 2024-03-29T11:33:17.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.627+0000 7fa7c4b34640 10 monclient: monmap: 2024-03-29T11:33:17.623 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:17.628443+0000 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:17.628443+0000 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:17.624 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.628+0000 7fa7c4b34640 1 Processor -- start 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.628+0000 7fa7c4b34640 1 -- start start 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.628+0000 7fa7c4b34640 10 monclient: init 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b802b000 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b80c1d40 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b80c4c00 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b802b3e0 con 0x7fa7b802b000 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b802c1f0 con 0x7fa7b80c1d40 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80c5050 con 0x7fa7b80c4c00 2024-03-29T11:33:17.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.629+0000 7fa7c4b34640 10 monclient(hunting): authenticate will time out at 851.783690s 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7bffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa7b80c4c00 0x7fa7b80d0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57772/0 (socket says 172.21.15.71:57772) 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7bffff640 1 -- 172.21.15.71:0/72533089 learned_addr learned my addr 172.21.15.71:0/72533089 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036470191 0 0) 0x7fa7b80c5050 con 0x7fa7b80c4c00 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7be7fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.630+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7b0003520 con 0x7fa7b80c4c00 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.631+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963933831 0 0) 0x7fa7b0003520 con 0x7fa7b80c4c00 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.631+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 >> v1:172.21.15.81:6789/0 conn(0x7fa7b80c1d40 legacy=0x7fa7b80c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:17.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.631+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 >> v1:172.21.15.71:6789/0 conn(0x7fa7b802b000 legacy=0x7fa7b80c1630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 1 monclient: found mon.noname-b 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80cadd0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:17.631 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa7c004f730 con 0x7fa7b80c4c00 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7c4b34640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7c4b34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: dump: 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:17.632 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7c4b34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7c00008d0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7c4b34640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7c4b34640 4 set_mon_vals no callback set 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 1 -- 172.21.15.71:0/72533089 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa7c00758b0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.632+0000 7fa7be7fc640 10 monclient: dump: 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:17.633 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.633+0000 7fa7c4b34640 10 monclient: shutdown 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.633+0000 7fa7c4b34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.633+0000 7fa7c4b34640 1 -- 172.21.15.71:0/72533089 >> v1:172.21.15.71:6790/0 conn(0x7fa7b80c4c00 legacy=0x7fa7b80d0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.633+0000 7fa7c4b34640 1 -- 172.21.15.71:0/72533089 shutdown_connections 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.633+0000 7fa7c4b34640 1 -- 172.21.15.71:0/72533089 >> 172.21.15.71:0/72533089 conn(0x7fa7b803a590 msgr2=0x7fa7b80367e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:17.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.634+0000 7fa7c4b34640 1 -- 172.21.15.71:0/72533089 shutdown_connections 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.634+0000 7fa7c4b34640 1 -- 172.21.15.71:0/72533089 wait complete. 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 10 monclient: build_initial_monmap 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 10 monclient: monmap: 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 1 librados: starting msgr at 2024-03-29T11:33:17.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 1 librados: starting objecter 2024-03-29T11:33:17.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 1 Processor -- start 2024-03-29T11:33:17.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 1 -- start start 2024-03-29T11:33:17.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 1 librados: setting wanted keys 2024-03-29T11:33:17.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 1 librados: calling monclient init 2024-03-29T11:33:17.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.635+0000 7fa7c4b34640 10 monclient: init 2024-03-29T11:33:17.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:17.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:17.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b802b000 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:17.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b80c1d40 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b80c4c00 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80c81e0 con 0x7fa7b80c1d40 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80b5d20 con 0x7fa7b802b000 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b80c99c0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.636+0000 7fa7c4b34640 10 monclient(hunting): authenticate will time out at 851.790897s 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.637+0000 7fa7bffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa7b80c4c00 0x7fa7b80b5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57864/0 (socket says 172.21.15.71:57864) 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.637+0000 7fa7bffff640 1 -- 172.21.15.71:0/2851945684 learned_addr learned my addr 172.21.15.71:0/2851945684 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.637+0000 7fa7abfff640 10 client.?.objecter ms_handle_connect 0x7fa7b80c4c00 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.637+0000 7fa7abfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538225031 0 0) 0x7fa7b80c99c0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:17.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7a4004eb0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 203871240 0 0) 0x7fa7a4004eb0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7b80c99c0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa7c0068890 con 0x7fa7b80c4c00 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): dump: 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.643 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065400666 0 0) 0x7fa7b80c99c0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 >> v1:172.21.15.81:6789/0 conn(0x7fa7b802b000 legacy=0x7fa7b80c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.638+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 >> v1:172.21.15.71:6789/0 conn(0x7fa7b80c1d40 legacy=0x7fa7b80b4f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 1 monclient: found mon.c 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b80c7d90 con 0x7fa7b80c4c00 2024-03-29T11:33:17.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 10 monclient: _finish_auth 0 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 10 monclient: _renew_subs 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b80c8670 con 0x7fa7b80c4c00 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 10 monclient: _renew_subs 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b80c7a20 con 0x7fa7b80c4c00 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.641+0000 7fa7c4b34640 1 librados: init done 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7c004fdd0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa7c0067bf0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7abfff640 10 monclient: dump: 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.642+0000 7fa7bdffb640 4 set_mon_vals no callback set 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fa7c0067e10 con 0x7fa7b80c4c00 2024-03-29T11:33:17.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fa7c00ff7f0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7abfff640 10 client.4283.objecter ms_dispatch 0x7fa7c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7abfff640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7abfff640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7c4b34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7c4b34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7abfff640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7c4b34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.643+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a0005080 con 0x7fa7b80c4c00 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.644+0000 7fa7bf7fe640 10 monclient: get_auth_request con 0x7fa7a406db40 auth_method 0 2024-03-29T11:33:17.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.644+0000 7fa7abfff640 10 client.4283.objecter ms_handle_connect 0x7fa7a406db40 2024-03-29T11:33:17.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.653+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7c00cb0d0 con 0x7fa7b80c4c00 2024-03-29T11:33:17.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.653+0000 7fa7abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:17.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.653+0000 7fa7abfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:17.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.973+0000 7fa7c4b34640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:17.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.973+0000 7fa7c4b34640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:17.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.973+0000 7fa7c4b34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:17.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.973+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7a0005330 con 0x7fa7b80c4c00 2024-03-29T11:33:17.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.974+0000 7fa7abfff640 1 -- 172.21.15.71:0/2851945684 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fa7c00cd000 con 0x7fa7b80c4c00 2024-03-29T11:33:17.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.974+0000 7fa7abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:17.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.974+0000 7fa7abfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:17.970 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:17.971 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.976+0000 7fa7c4b34640 10 librados: watch_flush enter 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 10 librados: watch_flush exit 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 >> v1:172.21.15.71:6816/1596514235 conn(0x7fa7a406db40 legacy=0x7fa7a4070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 10 monclient: shutdown 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 >> v1:172.21.15.71:6790/0 conn(0x7fa7b80c4c00 legacy=0x7fa7b80b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:17.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 shutdown_connections 2024-03-29T11:33:17.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 >> 172.21.15.71:0/2851945684 conn(0x7fa7b803a590 msgr2=0x7fa7b80c1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:17.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.977+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 shutdown_connections 2024-03-29T11:33:17.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.978+0000 7fa7c4b34640 1 -- 172.21.15.71:0/2851945684 wait complete. 2024-03-29T11:33:17.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:17.978+0000 7fa7c4b34640 1 librados: shutdown 2024-03-29T11:33:17.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:24.795 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:24.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.929+0000 7f40780a8640 10 monclient: get_monmap_and_config 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.929+0000 7f40780a8640 10 monclient: build_initial_monmap 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.929+0000 7f40780a8640 10 monclient: monmap: 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:24.930649+0000 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:24.930649+0000 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:24.926 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.930+0000 7f40780a8640 1 Processor -- start 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.930+0000 7f40780a8640 1 -- start start 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.930+0000 7f40780a8640 10 monclient: init 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient(hunting): picked mon.noname-a con 0x7f40701457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient(hunting): picked mon.noname-c con 0x7f407013f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient(hunting): picked mon.noname-b con 0x7f40700a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40700aaef0 con 0x7f40701457f0 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4070145bd0 con 0x7f407013f650 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f4076645640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f40700a9d70 0x7f407014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56804/0 (socket says 172.21.15.71:56804) 2024-03-29T11:33:24.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f4076645640 1 -- 172.21.15.71:0/2852670270 learned_addr learned my addr 172.21.15.71:0/2852670270 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 1 -- 172.21.15.71:0/2852670270 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4070145eb0 con 0x7f40700a9d70 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.931+0000 7f40780a8640 10 monclient(hunting): authenticate will time out at 859.085965s 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726249139 0 0) 0x7f4070145eb0 con 0x7f40700a9d70 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 10 monclient(hunting): _negotiate 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c003520 con 0x7f40700a9d70 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432098625 0 0) 0x7f405c003520 con 0x7f40700a9d70 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 >> v1:172.21.15.81:6789/0 conn(0x7f407013f650 legacy=0x7f40700b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.932+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 >> v1:172.21.15.71:6789/0 conn(0x7f40701457f0 legacy=0x7f407013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: _finish_hunting 0 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 1 monclient: found mon.noname-b 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4070146340 con 0x7f40700a9d70 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: _finish_auth 0 2024-03-29T11:33:24.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: _check_auth_tickets 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4068001a10 con 0x7f40700a9d70 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: dump: 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:24.932 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f40780a8640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f40780a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4068001be0 con 0x7f40700a9d70 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f40780a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f40780a8640 4 set_mon_vals no callback set 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 1 -- 172.21.15.71:0/2852670270 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4068002030 con 0x7f40700a9d70 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.935+0000 7f4074e42640 10 monclient: dump: 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.933 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.934 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:24.934 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:24.934 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:24.934 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:24.934 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:24.934 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.936+0000 7f40780a8640 10 monclient: shutdown 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.936+0000 7f40780a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.936+0000 7f40780a8640 1 -- 172.21.15.71:0/2852670270 >> v1:172.21.15.71:6790/0 conn(0x7f40700a9d70 legacy=0x7f407014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.938+0000 7f40780a8640 1 -- 172.21.15.71:0/2852670270 shutdown_connections 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.938+0000 7f40780a8640 1 -- 172.21.15.71:0/2852670270 >> 172.21.15.71:0/2852670270 conn(0x7f4070066a00 msgr2=0x7f4070066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.938+0000 7f40780a8640 1 -- 172.21.15.71:0/2852670270 shutdown_connections 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.938+0000 7f40780a8640 1 -- 172.21.15.71:0/2852670270 wait complete. 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.938+0000 7f40780a8640 10 monclient: build_initial_monmap 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.938+0000 7f40780a8640 10 monclient: monmap: 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:24.937 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 1 librados: starting msgr at 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 1 librados: starting objecter 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 1 Processor -- start 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 1 -- start start 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 1 librados: setting wanted keys 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 1 librados: calling monclient init 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.939+0000 7f40780a8640 10 monclient: init 2024-03-29T11:33:24.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 10 monclient(hunting): picked mon.noname-a con 0x7f40700a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 10 monclient(hunting): picked mon.noname-c con 0x7f407013f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 10 monclient(hunting): picked mon.noname-b con 0x7f40701457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4070070860 con 0x7f40700a9d70 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f4075643640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f407013f650 0x7f4070070cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57784/0 (socket says 172.21.15.71:57784) 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f4075643640 1 -- 172.21.15.71:0/2438010840 learned_addr learned my addr 172.21.15.71:0/2438010840 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.940+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4070071400 con 0x7f40701457f0 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40700b17a0 con 0x7f407013f650 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40780a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:24.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40780a8640 10 monclient(hunting): authenticate will time out at 859.095361s 2024-03-29T11:33:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 client.?.objecter ms_handle_connect 0x7f407013f650 2024-03-29T11:33:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 client.?.objecter ms_handle_connect 0x7f40701457f0 2024-03-29T11:33:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457642410 0 0) 0x7f40700b17a0 con 0x7f407013f650 2024-03-29T11:33:24.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.941+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4054004eb0 con 0x7f407013f650 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1010610001 0 0) 0x7f4054004eb0 con 0x7f407013f650 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40700b17a0 con 0x7f407013f650 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4060001a30 con 0x7f407013f650 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): dump: 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.944 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364592552 0 0) 0x7f4070071400 con 0x7f40701457f0 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4054004eb0 con 0x7f40701457f0 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2614140849 0 0) 0x7f40700b17a0 con 0x7f407013f650 2024-03-29T11:33:24.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 >> v1:172.21.15.81:6789/0 conn(0x7f40701457f0 legacy=0x7f407013a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.942+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 >> v1:172.21.15.71:6789/0 conn(0x7f40700a9d70 legacy=0x7f407006ff70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 1 monclient: found mon.c 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40701deab0 con 0x7f407013f650 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4060001b90 con 0x7f407013f650 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.943+0000 7f40667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4060003a60 con 0x7f407013f650 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:24.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40667fc640 10 monclient: dump: 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40780a8640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40780a8640 10 monclient: _renew_subs 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40780a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40701ded60 con 0x7f407013f650 2024-03-29T11:33:24.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.944+0000 7f40677fe640 4 set_mon_vals no callback set 2024-03-29T11:33:24.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.946+0000 7f40780a8640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:24.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.946+0000 7f40780a8640 10 monclient: _renew_subs 2024-03-29T11:33:24.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.946+0000 7f40780a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:24.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.946+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f407013afa0 con 0x7f407013f650 2024-03-29T11:33:24.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.946+0000 7f40780a8640 1 librados: init done 2024-03-29T11:33:24.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.947+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f4060003bc0 con 0x7f407013f650 2024-03-29T11:33:24.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.947+0000 7f40780a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:24.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.947+0000 7f40780a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:24.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.947+0000 7f40780a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:24.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.947+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407013b250 con 0x7f407013f650 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.948+0000 7f4075e44640 10 monclient: get_auth_request con 0x7f405406de40 auth_method 0 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.948+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f4060084b20 con 0x7f407013f650 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.948+0000 7f40667fc640 10 client.4289.objecter ms_dispatch 0x7f4070000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.948+0000 7f40667fc640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.948+0000 7f40667fc640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.949+0000 7f40667fc640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:33:24.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.949+0000 7f40667fc640 10 client.4289.objecter ms_handle_connect 0x7f405406de40 2024-03-29T11:33:24.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.955+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4060050460 con 0x7f407013f650 2024-03-29T11:33:24.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.955+0000 7f40667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:24.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:24.955+0000 7f40667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:25.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.252+0000 7f40780a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:25.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.252+0000 7f40780a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:25.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.252+0000 7f40780a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:25.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.252+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4070070a10 con 0x7f407013f650 2024-03-29T11:33:25.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.253+0000 7f40667fc640 1 -- 172.21.15.71:0/2438010840 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f4070070a10 con 0x7f407013f650 2024-03-29T11:33:25.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.253+0000 7f40667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:25.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.253+0000 7f40667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:25.250 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:25.250 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:25.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.256+0000 7f40780a8640 10 librados: watch_flush enter 2024-03-29T11:33:25.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.256+0000 7f40780a8640 10 librados: watch_flush exit 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.256+0000 7f40780a8640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 >> v1:172.21.15.71:6816/1596514235 conn(0x7f405406de40 legacy=0x7f4054070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 10 monclient: shutdown 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 >> v1:172.21.15.71:6790/0 conn(0x7f407013f650 legacy=0x7f4070070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 shutdown_connections 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 >> 172.21.15.71:0/2438010840 conn(0x7f4070066a00 msgr2=0x7f407013dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 shutdown_connections 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 -- 172.21.15.71:0/2438010840 wait complete. 2024-03-29T11:33:25.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:25.257+0000 7f40780a8640 1 librados: shutdown 2024-03-29T11:33:25.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:32.075 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.231+0000 7fd947fff640 10 monclient: get_monmap_and_config 2024-03-29T11:33:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.231+0000 7fd947fff640 10 monclient: build_initial_monmap 2024-03-29T11:33:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.232+0000 7fd947fff640 10 monclient: monmap: 2024-03-29T11:33:32.228 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:32.228 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:32.228 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:32.232925+0000 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:32.232925+0000 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:32.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 1 Processor -- start 2024-03-29T11:33:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 1 -- start start 2024-03-29T11:33:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 10 monclient: init 2024-03-29T11:33:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd9400389e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.233+0000 7fd947fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd940038dc0 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.234+0000 7fd947fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd940030660 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.234+0000 7fd947fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9400c54c0 con 0x7fd940030660 2024-03-29T11:33:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.234+0000 7fd947fff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd940036430 con 0x7fd940038dc0 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.234+0000 7fd947fff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd940039840 con 0x7fd9400389e0 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.234+0000 7fd947fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.234+0000 7fd947fff640 10 monclient(hunting): authenticate will time out at 866.388104s 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd9477fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd940030660 0x7fd9400d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43901/0 (socket says 172.21.15.71:43901) 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd9477fe640 1 -- 172.21.15.71:0/3657957950 learned_addr learned my addr 172.21.15.71:0/3657957950 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973407746 0 0) 0x7fd9400c54c0 con 0x7fd940030660 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd945ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd945ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd945ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd945ffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:33:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.236+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd930003520 con 0x7fd940030660 2024-03-29T11:33:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183464339 0 0) 0x7fd940039840 con 0x7fd9400389e0 2024-03-29T11:33:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:33:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9400c54c0 con 0x7fd9400389e0 2024-03-29T11:33:32.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.240+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496295023 0 0) 0x7fd9400c54c0 con 0x7fd9400389e0 2024-03-29T11:33:32.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 >> v1:172.21.15.81:6789/0 conn(0x7fd940038dc0 legacy=0x7fd9400cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 >> v1:172.21.15.71:6789/0 conn(0x7fd940030660 legacy=0x7fd9400d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:32.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 monclient: found mon.noname-b 2024-03-29T11:33:32.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd940035690 con 0x7fd9400389e0 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd94804f730 con 0x7fd9400389e0 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: dump: 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.238 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd948075f10 con 0x7fd9400389e0 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 1 -- 172.21.15.71:0/3657957950 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd9480758b0 con 0x7fd9400389e0 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd945ffb640 10 monclient: dump: 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:32.239 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd947fff640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd947fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:32.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.241+0000 7fd947fff640 4 set_mon_vals no callback set 2024-03-29T11:33:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.242+0000 7fd947fff640 10 monclient: shutdown 2024-03-29T11:33:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.242+0000 7fd947fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.242+0000 7fd947fff640 1 -- 172.21.15.71:0/3657957950 >> v1:172.21.15.71:6790/0 conn(0x7fd9400389e0 legacy=0x7fd940033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.243+0000 7fd947fff640 1 -- 172.21.15.71:0/3657957950 shutdown_connections 2024-03-29T11:33:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.243+0000 7fd947fff640 1 -- 172.21.15.71:0/3657957950 >> 172.21.15.71:0/3657957950 conn(0x7fd940005c00 msgr2=0x7fd940006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.243+0000 7fd947fff640 1 -- 172.21.15.71:0/3657957950 shutdown_connections 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.243+0000 7fd947fff640 1 -- 172.21.15.71:0/3657957950 wait complete. 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.244+0000 7fd947fff640 10 monclient: build_initial_monmap 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.244+0000 7fd947fff640 10 monclient: monmap: 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.244+0000 7fd947fff640 1 librados: starting msgr at 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.244+0000 7fd947fff640 1 librados: starting objecter 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.245+0000 7fd947fff640 1 Processor -- start 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.245+0000 7fd947fff640 1 -- start start 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.245+0000 7fd947fff640 1 librados: setting wanted keys 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.245+0000 7fd947fff640 1 librados: calling monclient init 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.245+0000 7fd947fff640 10 monclient: init 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd940030660 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd9400389e0 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient(hunting): picked mon.noname-a con 0x7fd9401608e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94015fd40 con 0x7fd9401608e0 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9400c7150 con 0x7fd9400389e0 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd94002fa00 con 0x7fd940030660 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.246+0000 7fd947fff640 10 monclient(hunting): authenticate will time out at 866.400472s 2024-03-29T11:33:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd946ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd940030660 0x7fd9400c58c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:41063/0 (socket says 172.21.15.71:41063) 2024-03-29T11:33:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd946ffd640 1 -- 172.21.15.71:0/2402641822 learned_addr learned my addr 172.21.15.71:0/2402641822 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 10 client.?.objecter ms_handle_connect 0x7fd940030660 2024-03-29T11:33:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356954992 0 0) 0x7fd94002fa00 con 0x7fd940030660 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.251+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd934004eb0 con 0x7fd940030660 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1978100217 0 0) 0x7fd934004eb0 con 0x7fd940030660 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd94002fa00 con 0x7fd940030660 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd94804f400 con 0x7fd940030660 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 10 monclient(hunting): dump: 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:32.250 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.252+0000 7fd92f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.253+0000 7fd9477fe640 1 -- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6789/0 conn(0x7fd9401608e0 legacy=0x7fd9400c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.253+0000 7fd9477fe640 1 -- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6789/0 conn(0x7fd9401608e0 legacy=0x7fd9400c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.253+0000 7fd9477fe640 1 --1- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6789/0 conn(0x7fd9401608e0 0x7fd9400c7cd0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.253+0000 7fd9477fe640 1 --1- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6789/0 conn(0x7fd9401608e0 0x7fd9400c7cd0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-29T11:33:32.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.253+0000 7fd92f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.71:6789/0 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.255+0000 7fd92f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9400389e0 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.255+0000 7fd92f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 677765165 0 0) 0x7fd94002fa00 con 0x7fd940030660 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 >> v1:172.21.15.81:6789/0 conn(0x7fd9400389e0 legacy=0x7fd9401601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6789/0 conn(0x7fd9401608e0 legacy=0x7fd9400c7cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 monclient: found mon.c 2024-03-29T11:33:32.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9400c7500 con 0x7fd940030660 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd94804f730 con 0x7fd940030660 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd948067d90 con 0x7fd940030660 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.256+0000 7fd92f7fe640 10 monclient: dump: 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:32.256 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 10 monclient: _renew_subs 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9400c7750 con 0x7fd940030660 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 10 monclient: _renew_subs 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:32.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9400c85c0 con 0x7fd940030660 2024-03-29T11:33:32.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd947fff640 1 librados: init done 2024-03-29T11:33:32.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.257+0000 7fd9457fa640 4 set_mon_vals no callback set 2024-03-29T11:33:32.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.258+0000 7fd947fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:32.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.258+0000 7fd947fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:32.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.258+0000 7fd947fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:32.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.258+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd928005080 con 0x7fd940030660 2024-03-29T11:33:32.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.263+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fd948067ef0 con 0x7fd940030660 2024-03-29T11:33:32.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.263+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fd9480ff660 con 0x7fd940030660 2024-03-29T11:33:32.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.264+0000 7fd9467fc640 10 monclient: get_auth_request con 0x7fd9340763c0 auth_method 0 2024-03-29T11:33:32.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.264+0000 7fd92f7fe640 10 client.4295.objecter ms_dispatch 0x7fd948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:32.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.264+0000 7fd92f7fe640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:32.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.264+0000 7fd92f7fe640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:32.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.264+0000 7fd92f7fe640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:33:32.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.264+0000 7fd92f7fe640 10 client.4295.objecter ms_handle_connect 0x7fd9340763c0 2024-03-29T11:33:32.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.269+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9480caf40 con 0x7fd940030660 2024-03-29T11:33:32.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.270+0000 7fd92f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:32.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.270+0000 7fd92f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:32.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.591+0000 7fd947fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:32.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.591+0000 7fd947fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:32.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.591+0000 7fd947fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:32.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.591+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd928005330 con 0x7fd940030660 2024-03-29T11:33:32.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.596+0000 7fd92f7fe640 1 -- 172.21.15.71:0/2402641822 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fd9480cce70 con 0x7fd940030660 2024-03-29T11:33:32.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.596+0000 7fd92f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:32.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.596+0000 7fd92f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:32.594 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:32.594 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:32.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.600+0000 7fd947fff640 10 librados: watch_flush enter 2024-03-29T11:33:32.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 10 librados: watch_flush exit 2024-03-29T11:33:32.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:33:32.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:33:32.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6816/1596514235 conn(0x7fd9340763c0 legacy=0x7fd934078880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 10 monclient: shutdown 2024-03-29T11:33:32.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:32.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 >> v1:172.21.15.71:6790/0 conn(0x7fd940030660 legacy=0x7fd9400c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:32.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 shutdown_connections 2024-03-29T11:33:32.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.602+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 >> 172.21.15.71:0/2402641822 conn(0x7fd940005c00 msgr2=0x7fd940005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:32.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.603+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 shutdown_connections 2024-03-29T11:33:32.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.603+0000 7fd947fff640 1 -- 172.21.15.71:0/2402641822 wait complete. 2024-03-29T11:33:32.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:32.603+0000 7fd947fff640 1 librados: shutdown 2024-03-29T11:33:32.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:39.423 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.573+0000 7f6b33fff640 10 monclient: get_monmap_and_config 2024-03-29T11:33:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.573+0000 7f6b33fff640 10 monclient: build_initial_monmap 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.573+0000 7f6b33fff640 10 monclient: monmap: 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:39.574127+0000 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:39.574127+0000 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:39.570 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:39.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.573+0000 7f6b33fff640 1 Processor -- start 2024-03-29T11:33:39.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.574+0000 7f6b33fff640 1 -- start start 2024-03-29T11:33:39.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.574+0000 7f6b33fff640 10 monclient: init 2024-03-29T11:33:39.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.574+0000 7f6b33fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:39.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.574+0000 7f6b33fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:39.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.574+0000 7f6b33fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6b340b2800 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:39.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 10 monclient(hunting): picked mon.noname-b con 0x7f6b340a9c00 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:39.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 10 monclient(hunting): picked mon.noname-a con 0x7f6b340b38c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b34141eb0 con 0x7f6b340b38c0 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b340b22d0 con 0x7f6b340b2800 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b34142d30 con 0x7f6b340a9c00 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.575+0000 7f6b33fff640 10 monclient(hunting): authenticate will time out at 873.729105s 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.578+0000 7f6b32ffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6b340b2800 0x7f6b34142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:54629/0 (socket says 172.21.15.71:54629) 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.578+0000 7f6b32ffd640 1 -- 172.21.15.71:0/2183761797 learned_addr learned my addr 172.21.15.71:0/2183761797 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.578+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020271830 0 0) 0x7f6b340b22d0 con 0x7f6b340b2800 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.579+0000 7f6b31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.579+0000 7f6b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.579+0000 7f6b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.579+0000 7f6b31ffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:33:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.579+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b1c003520 con 0x7f6b340b2800 2024-03-29T11:33:39.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1818361507 0 0) 0x7f6b1c003520 con 0x7f6b340b2800 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 >> v1:172.21.15.71:6790/0 conn(0x7f6b340a9c00 legacy=0x7f6b3413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 >> v1:172.21.15.71:6789/0 conn(0x7f6b340b38c0 legacy=0x7f6b3414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 1 monclient: found mon.noname-c 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.582+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b340aaf90 con 0x7f6b340b2800 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b33fff640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b33fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6b24001690 con 0x7f6b340b2800 2024-03-29T11:33:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.583+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 10 monclient: dump: 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:39.582 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b33fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b240018b0 con 0x7f6b340b2800 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b33fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b33fff640 4 set_mon_vals no callback set 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 1 -- 172.21.15.71:0/2183761797 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6b24001c80 con 0x7f6b340b2800 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.584+0000 7f6b31ffb640 10 monclient: dump: 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.583 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 10 monclient: shutdown 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 1 -- 172.21.15.71:0/2183761797 >> v1:172.21.15.81:6789/0 conn(0x7f6b340b2800 legacy=0x7f6b34142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 1 -- 172.21.15.71:0/2183761797 shutdown_connections 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 1 -- 172.21.15.71:0/2183761797 >> 172.21.15.71:0/2183761797 conn(0x7f6b34066a00 msgr2=0x7f6b34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:39.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 1 -- 172.21.15.71:0/2183761797 shutdown_connections 2024-03-29T11:33:39.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.585+0000 7f6b33fff640 1 -- 172.21.15.71:0/2183761797 wait complete. 2024-03-29T11:33:39.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.589+0000 7f6b33fff640 10 monclient: build_initial_monmap 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.589+0000 7f6b33fff640 10 monclient: monmap: 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:39.586 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:39.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.590+0000 7f6b33fff640 1 librados: starting msgr at 2024-03-29T11:33:39.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.590+0000 7f6b33fff640 1 librados: starting objecter 2024-03-29T11:33:39.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.590+0000 7f6b33fff640 1 Processor -- start 2024-03-29T11:33:39.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.591+0000 7f6b33fff640 1 -- start start 2024-03-29T11:33:39.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.591+0000 7f6b33fff640 1 librados: setting wanted keys 2024-03-29T11:33:39.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.591+0000 7f6b33fff640 1 librados: calling monclient init 2024-03-29T11:33:39.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.591+0000 7f6b33fff640 10 monclient: init 2024-03-29T11:33:39.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.591+0000 7f6b33fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:39.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.591+0000 7f6b33fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:39.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6b340a9c00 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 10 monclient(hunting): picked mon.noname-b con 0x7f6b340b2800 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 10 monclient(hunting): picked mon.noname-a con 0x7f6b340b38c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b34070800 con 0x7f6b340b38c0 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b341dedd0 con 0x7f6b340b2800 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3406ef30 con 0x7f6b340a9c00 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.594+0000 7f6b33fff640 10 monclient(hunting): authenticate will time out at 873.748712s 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.595+0000 7f6b32ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f6b340a9c00 0x7f6b341de6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47505/0 (socket says 172.21.15.71:47505) 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.595+0000 7f6b32ffd640 1 -- 172.21.15.71:0/1949337923 learned_addr learned my addr 172.21.15.71:0/1949337923 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.595+0000 7f6b1b7fe640 10 client.?.objecter ms_handle_connect 0x7f6b340a9c00 2024-03-29T11:33:39.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.595+0000 7f6b1b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.595+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389482536 0 0) 0x7f6b3406ef30 con 0x7f6b340a9c00 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b20004eb0 con 0x7f6b340a9c00 2024-03-29T11:33:39.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326797738 0 0) 0x7f6b20004eb0 con 0x7f6b340a9c00 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b3406ef30 con 0x7f6b340a9c00 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6b24001550 con 0x7f6b340a9c00 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): dump: 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:39.595 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:39.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:39.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.596+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1174860312 0 0) 0x7f6b3406ef30 con 0x7f6b340a9c00 2024-03-29T11:33:39.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.597+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 >> v1:172.21.15.81:6789/0 conn(0x7f6b340b2800 legacy=0x7f6b340b09a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 >> v1:172.21.15.71:6789/0 conn(0x7f6b340b38c0 legacy=0x7f6b340b10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 1 monclient: found mon.c 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3406f0c0 con 0x7f6b340a9c00 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b33fff640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b33fff640 10 monclient: _renew_subs 2024-03-29T11:33:39.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b33fff640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b34070d20 con 0x7f6b340a9c00 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b240016b0 con 0x7f6b340a9c00 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6b24004e90 con 0x7f6b340a9c00 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b1b7fe640 10 monclient: dump: 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:39.600 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.599+0000 7f6b317fa640 4 set_mon_vals no callback set 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.600+0000 7f6b33fff640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.600+0000 7f6b33fff640 10 monclient: _renew_subs 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.600+0000 7f6b33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.600+0000 7f6b33fff640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3413ec60 con 0x7f6b340a9c00 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.600+0000 7f6b33fff640 1 librados: init done 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.601+0000 7f6b197fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.601+0000 7f6b197fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.601+0000 7f6b197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:39.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.601+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c005080 con 0x7f6b340a9c00 2024-03-29T11:33:39.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.602+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f6b2401ae60 con 0x7f6b340a9c00 2024-03-29T11:33:39.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.603+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f6b24054c60 con 0x7f6b340a9c00 2024-03-29T11:33:39.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.603+0000 7f6b1b7fe640 10 client.4298.objecter ms_dispatch 0x7f6b34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:39.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.603+0000 7f6b1b7fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:39.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.603+0000 7f6b1b7fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:39.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.603+0000 7f6b1b7fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:33:39.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.604+0000 7f6b327fc640 10 monclient: get_auth_request con 0x7f6b2006db80 auth_method 0 2024-03-29T11:33:39.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.604+0000 7f6b1b7fe640 10 client.4298.objecter ms_handle_connect 0x7f6b2006db80 2024-03-29T11:33:39.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.612+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b24050f00 con 0x7f6b340a9c00 2024-03-29T11:33:39.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.612+0000 7f6b1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:39.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.612+0000 7f6b1b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:39.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b197fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:39.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b197fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:39.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:39.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b0c005330 con 0x7f6b340a9c00 2024-03-29T11:33:39.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b1b7fe640 1 -- 172.21.15.71:0/1949337923 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f6b240510a0 con 0x7f6b340a9c00 2024-03-29T11:33:39.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:39.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.929+0000 7f6b1b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:39.929 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:39.929 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:39.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 10 librados: watch_flush enter 2024-03-29T11:33:39.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 10 librados: watch_flush exit 2024-03-29T11:33:39.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:33:39.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 >> v1:172.21.15.71:6816/1596514235 conn(0x7f6b2006db80 legacy=0x7f6b20070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 10 monclient: shutdown 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 >> v1:172.21.15.71:6790/0 conn(0x7f6b340a9c00 legacy=0x7f6b341de6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 shutdown_connections 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.935+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 >> 172.21.15.71:0/1949337923 conn(0x7f6b34066a00 msgr2=0x7f6b340b7b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.936+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 shutdown_connections 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.936+0000 7f6b197fa640 1 -- 172.21.15.71:0/1949337923 wait complete. 2024-03-29T11:33:39.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:39.937+0000 7f6b197fa640 1 librados: shutdown 2024-03-29T11:33:39.951 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:46.758 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:46.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.921+0000 7f58c3dfd640 10 monclient: get_monmap_and_config 2024-03-29T11:33:46.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.921+0000 7f58c3dfd640 10 monclient: build_initial_monmap 2024-03-29T11:33:46.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.921+0000 7f58c3dfd640 10 monclient: monmap: 2024-03-29T11:33:46.918 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:46.922625+0000 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:46.922625+0000 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:46.919 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:46.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.924+0000 7f58c3dfd640 1 Processor -- start 2024-03-29T11:33:46.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.926+0000 7f58c3dfd640 1 -- start start 2024-03-29T11:33:46.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.926+0000 7f58c3dfd640 10 monclient: init 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.927+0000 7f58c3dfd640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.927+0000 7f58c3dfd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.927+0000 7f58c3dfd640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc1457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.927+0000 7f58c3dfd640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.928+0000 7f58c3dfd640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.928+0000 7f58c3dfd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0aaef0 con 0x7f58bc0a9d70 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.928+0000 7f58c3dfd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc145bd0 con 0x7f58bc1457f0 2024-03-29T11:33:46.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.928+0000 7f58c3dfd640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc145eb0 con 0x7f58bc13f650 2024-03-29T11:33:46.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.928+0000 7f58c3dfd640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:46.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.928+0000 7f58c3dfd640 10 monclient(hunting): authenticate will time out at 881.082173s 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.931+0000 7f58c1b99640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f58bc1457f0 0x7f58bc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:46391/0 (socket says 172.21.15.71:46391) 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.931+0000 7f58c1b99640 1 -- 172.21.15.71:0/677458424 learned_addr learned my addr 172.21.15.71:0/677458424 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.932+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499481537 0 0) 0x7f58bc145eb0 con 0x7f58bc13f650 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.932+0000 7f58c0b97640 10 monclient(hunting): _negotiate 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.932+0000 7f58c0b97640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.932+0000 7f58c0b97640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.932+0000 7f58c0b97640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.932+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a8003520 con 0x7f58bc13f650 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727820798 0 0) 0x7f58bc145bd0 con 0x7f58bc1457f0 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient(hunting): _negotiate 2024-03-29T11:33:46.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58bc145eb0 con 0x7f58bc1457f0 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2314434307 0 0) 0x7f58a8003520 con 0x7f58bc13f650 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 >> v1:172.21.15.81:6789/0 conn(0x7f58bc1457f0 legacy=0x7f58bc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 >> v1:172.21.15.71:6789/0 conn(0x7f58bc0a9d70 legacy=0x7f58bc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: _finish_hunting 0 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 monclient: found mon.noname-b 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc146340 con 0x7f58bc13f650 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: _finish_auth 0 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: _check_auth_tickets 2024-03-29T11:33:46.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f58b40018c0 con 0x7f58bc13f650 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: dump: 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:46.933 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b4001aa0 con 0x7f58bc13f650 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 1 -- 172.21.15.71:0/677458424 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f58b40018c0 con 0x7f58bc13f650 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.933+0000 7f58c0b97640 10 monclient: dump: 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:46.934 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:46.935 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:46.935 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:46.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.934+0000 7f58c3dfd640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:33:46.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.934+0000 7f58c3dfd640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:46.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.934+0000 7f58c3dfd640 4 set_mon_vals no callback set 2024-03-29T11:33:46.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.935+0000 7f58c3dfd640 10 monclient: shutdown 2024-03-29T11:33:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.935+0000 7f58c3dfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.935+0000 7f58c3dfd640 1 -- 172.21.15.71:0/677458424 >> v1:172.21.15.71:6790/0 conn(0x7f58bc13f650 legacy=0x7f58bc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.936+0000 7f58c3dfd640 1 -- 172.21.15.71:0/677458424 shutdown_connections 2024-03-29T11:33:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.936+0000 7f58c3dfd640 1 -- 172.21.15.71:0/677458424 >> 172.21.15.71:0/677458424 conn(0x7f58bc066a00 msgr2=0x7f58bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:46.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.938+0000 7f58c3dfd640 1 -- 172.21.15.71:0/677458424 shutdown_connections 2024-03-29T11:33:46.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.938+0000 7f58c3dfd640 1 -- 172.21.15.71:0/677458424 wait complete. 2024-03-29T11:33:46.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.939+0000 7f58c3dfd640 10 monclient: build_initial_monmap 2024-03-29T11:33:46.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.939+0000 7f58c3dfd640 10 monclient: monmap: 2024-03-29T11:33:46.938 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:46.938 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:46.938 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.939+0000 7f58c3dfd640 1 librados: starting msgr at 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.939+0000 7f58c3dfd640 1 librados: starting objecter 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 1 Processor -- start 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 1 -- start start 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 1 librados: setting wanted keys 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 1 librados: calling monclient init 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 10 monclient: init 2024-03-29T11:33:46.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:46.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:46.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc0a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:46.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.940+0000 7f58c3dfd640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc13f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:46.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.944+0000 7f58c3dfd640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.944+0000 7f58c3dfd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc1419e0 con 0x7f58bc13f650 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.944+0000 7f58c3dfd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc13fda0 con 0x7f58bc0a9d70 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.944+0000 7f58c3dfd640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc06f280 con 0x7f58bc1457f0 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.944+0000 7f58c3dfd640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.944+0000 7f58c3dfd640 10 monclient(hunting): authenticate will time out at 881.098393s 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.945+0000 7f58c1398640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f58bc13f650 0x7f58bc1332a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:52315/0 (socket says 172.21.15.71:52315) 2024-03-29T11:33:46.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.945+0000 7f58c1398640 1 -- 172.21.15.71:0/2494914128 learned_addr learned my addr 172.21.15.71:0/2494914128 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:46.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 client.?.objecter ms_handle_connect 0x7f58bc13f650 2024-03-29T11:33:46.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:46.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 client.?.objecter ms_handle_connect 0x7f58bc1457f0 2024-03-29T11:33:46.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898195652 0 0) 0x7f58bc1419e0 con 0x7f58bc13f650 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a0004eb0 con 0x7f58bc13f650 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149678383 0 0) 0x7f58bc06f280 con 0x7f58bc1457f0 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.946+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58bc1419e0 con 0x7f58bc1457f0 2024-03-29T11:33:46.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044546782 0 0) 0x7f58bc1419e0 con 0x7f58bc1457f0 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58bc06f280 con 0x7f58bc1457f0 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f58b8001720 con 0x7f58bc1457f0 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 10 monclient(hunting): dump: 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:46.949 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 305592936 0 0) 0x7f58bc06f280 con 0x7f58bc1457f0 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.947+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 >> v1:172.21.15.81:6789/0 conn(0x7f58bc0a9d70 legacy=0x7f58bc142760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 >> v1:172.21.15.71:6789/0 conn(0x7f58bc13f650 legacy=0x7f58bc1332a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 1 monclient: found mon.c 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc06f560 con 0x7f58bc1457f0 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:46.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b8001880 con 0x7f58bc1457f0 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f58b80035f0 con 0x7f58bc1457f0 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58b27fc640 10 monclient: dump: 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:46.951 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 10 monclient: _renew_subs 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc1339b0 con 0x7f58bc1457f0 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 10 monclient: _renew_subs 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc06e100 con 0x7f58bc1457f0 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.948+0000 7f58c3dfd640 1 librados: init done 2024-03-29T11:33:46.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.949+0000 7f58b3fff640 4 set_mon_vals no callback set 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.952+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f58b80195c0 con 0x7f58bc1457f0 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.952+0000 7f58c3dfd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.953+0000 7f58c3dfd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.953+0000 7f58c3dfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.953+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc075f10 con 0x7f58bc1457f0 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.955+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f58b80836b0 con 0x7f58bc1457f0 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.955+0000 7f58b27fc640 10 client.4304.objecter ms_dispatch 0x7f58bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.955+0000 7f58b27fc640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.955+0000 7f58b27fc640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.955+0000 7f58b27fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.956+0000 7f58c1b99640 10 monclient: get_auth_request con 0x7f58a006df60 auth_method 0 2024-03-29T11:33:46.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.957+0000 7f58b27fc640 10 client.4304.objecter ms_handle_connect 0x7f58a006df60 2024-03-29T11:33:46.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.963+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58b804ef90 con 0x7f58bc1457f0 2024-03-29T11:33:46.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.963+0000 7f58b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:46.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:46.963+0000 7f58b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:47.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.280+0000 7f58c3dfd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:47.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.280+0000 7f58c3dfd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:47.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.280+0000 7f58c3dfd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:33:47.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.280+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58bc0b3680 con 0x7f58bc1457f0 2024-03-29T11:33:47.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.283+0000 7f58b27fc640 1 -- 172.21.15.71:0/2494914128 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f58b8050ec0 con 0x7f58bc1457f0 2024-03-29T11:33:47.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.284+0000 7f58b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:47.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.284+0000 7f58b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:47.282 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:47.282 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:47.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.289+0000 7f58c3dfd640 10 librados: watch_flush enter 2024-03-29T11:33:47.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.289+0000 7f58c3dfd640 10 librados: watch_flush exit 2024-03-29T11:33:47.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.289+0000 7f58c3dfd640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:33:47.286 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.289+0000 7f58c3dfd640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.290+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 >> v1:172.21.15.71:6816/1596514235 conn(0x7f58a006df60 legacy=0x7f58a0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.290+0000 7f58c3dfd640 10 monclient: shutdown 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.290+0000 7f58c3dfd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.290+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 >> v1:172.21.15.71:6790/0 conn(0x7f58bc1457f0 legacy=0x7f58bc1ffad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.290+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 shutdown_connections 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.290+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 >> 172.21.15.71:0/2494914128 conn(0x7f58bc066a00 msgr2=0x7f58bc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:47.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.291+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 shutdown_connections 2024-03-29T11:33:47.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.291+0000 7f58c3dfd640 1 -- 172.21.15.71:0/2494914128 wait complete. 2024-03-29T11:33:47.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:47.291+0000 7f58c3dfd640 1 librados: shutdown 2024-03-29T11:33:47.304 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:54.112 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:54.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.264+0000 7f554923f640 10 monclient: get_monmap_and_config 2024-03-29T11:33:54.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.264+0000 7f554923f640 10 monclient: build_initial_monmap 2024-03-29T11:33:54.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.264+0000 7f554923f640 10 monclient: monmap: 2024-03-29T11:33:54.263 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:33:54.266642+0000 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:33:54.266642+0000 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.265+0000 7f554923f640 1 Processor -- start 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.266+0000 7f554923f640 1 -- start start 2024-03-29T11:33:54.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.266+0000 7f554923f640 10 monclient: init 2024-03-29T11:33:54.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.267+0000 7f554923f640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:54.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.267+0000 7f554923f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:33:54.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 10 monclient(hunting): picked mon.noname-b con 0x7f553c034850 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 10 monclient(hunting): picked mon.noname-a con 0x7f553c02a640 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f5543fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f553c034850 0x7f553c0c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52157/0 (socket says 172.21.15.71:52157) 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 10 monclient(hunting): picked mon.noname-c con 0x7f553c0327d0 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c02c820 con 0x7f553c02a640 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c02a120 con 0x7f553c0327d0 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0c75b0 con 0x7f553c034850 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f554923f640 10 monclient(hunting): authenticate will time out at 888.425424s 2024-03-29T11:33:54.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.270+0000 7f5543fff640 1 -- 172.21.15.71:0/3023777857 learned_addr learned my addr 172.21.15.71:0/3023777857 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:54.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.271+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351111489 0 0) 0x7f553c0c75b0 con 0x7f553c034850 2024-03-29T11:33:54.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.271+0000 7f5542ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:54.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.271+0000 7f5542ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:54.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.271+0000 7f5542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:54.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.271+0000 7f5542ffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:33:54.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.271+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5534003520 con 0x7f553c034850 2024-03-29T11:33:54.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.273+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888510770 0 0) 0x7f5534003520 con 0x7f553c034850 2024-03-29T11:33:54.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.273+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 >> v1:172.21.15.81:6789/0 conn(0x7f553c0327d0 legacy=0x7f553c0cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:54.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 >> v1:172.21.15.71:6789/0 conn(0x7f553c02a640 legacy=0x7f553c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:54.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:54.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 1 monclient: found mon.noname-b 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c033110 con 0x7f553c034850 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: _finish_auth 0 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5538001070 con 0x7f553c034850 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f554923f640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f554923f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f5542ffd640 10 monclient: dump: 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:54.274 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.275+0000 7f554923f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.276+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5538001290 con 0x7f553c034850 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.276+0000 7f5542ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.276+0000 7f5542ffd640 1 -- 172.21.15.71:0/3023777857 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5538001620 con 0x7f553c034850 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.276+0000 7f554923f640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:54.275 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.276+0000 7f554923f640 4 set_mon_vals no callback set 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 10 monclient: shutdown 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f5542ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f5542ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f5542ffd640 10 monclient: dump: 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.277 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 1 -- 172.21.15.71:0/3023777857 >> v1:172.21.15.71:6790/0 conn(0x7f553c034850 legacy=0x7f553c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 1 -- 172.21.15.71:0/3023777857 shutdown_connections 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 1 -- 172.21.15.71:0/3023777857 >> 172.21.15.71:0/3023777857 conn(0x7f553c0058f0 msgr2=0x7f553c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 1 -- 172.21.15.71:0/3023777857 shutdown_connections 2024-03-29T11:33:54.278 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.277+0000 7f554923f640 1 -- 172.21.15.71:0/3023777857 wait complete. 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.278+0000 7f554923f640 10 monclient: build_initial_monmap 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.278+0000 7f554923f640 10 monclient: monmap: 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.279 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 1 librados: starting msgr at 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 1 librados: starting objecter 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 1 Processor -- start 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 1 -- start start 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 1 librados: setting wanted keys 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 1 librados: calling monclient init 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.279+0000 7f554923f640 10 monclient: init 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:54.280 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient(hunting): picked mon.noname-b con 0x7f553c02a640 addr v1:172.21.15.81:6789/0 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient(hunting): picked mon.noname-a con 0x7f553c0327d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient(hunting): picked mon.noname-c con 0x7f553c034850 addr v1:172.21.15.71:6790/0 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0c21d0 con 0x7f553c0327d0 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0c3780 con 0x7f553c02a640 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553c0c39f0 con 0x7f553c034850 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:54.281 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.280+0000 7f554923f640 10 monclient(hunting): authenticate will time out at 888.435728s 2024-03-29T11:33:54.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.284+0000 7f5543fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f553c02a640 0x7f553c0c40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:48533/0 (socket says 172.21.15.71:48533) 2024-03-29T11:33:54.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.284+0000 7f5543fff640 1 -- 172.21.15.71:0/3329914506 learned_addr learned my addr 172.21.15.71:0/3329914506 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:33:54.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.284+0000 7f5540ff9640 10 client.?.objecter ms_handle_connect 0x7f553c02a640 2024-03-29T11:33:54.282 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.284+0000 7f5540ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:54.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.285+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829319559 0 0) 0x7f553c0c3780 con 0x7f553c02a640 2024-03-29T11:33:54.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.285+0000 7f5540ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:54.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.285+0000 7f5540ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:54.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.285+0000 7f5540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:54.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.285+0000 7f5540ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:33:54.283 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.285+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5528004eb0 con 0x7f553c02a640 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 client.?.objecter ms_handle_connect 0x7f553c034850 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220459156 0 0) 0x7f553c0c39f0 con 0x7f553c034850 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:54.287 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f553c0c3780 con 0x7f553c034850 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553438575 0 0) 0x7f5528004eb0 con 0x7f553c02a640 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f553c0c39f0 con 0x7f553c02a640 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5538001400 con 0x7f553c02a640 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:54.288 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.289 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.289+0000 7f5540ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:54.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 10 client.?.objecter ms_handle_connect 0x7f553c0327d0 2024-03-29T11:33:54.292 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442234604 0 0) 0x7f553c0c21d0 con 0x7f553c0327d0 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.290+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5528004eb0 con 0x7f553c0327d0 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1785992641 0 0) 0x7f553c0c3780 con 0x7f553c034850 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f553c0c21d0 con 0x7f553c034850 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f554404fdd0 con 0x7f553c034850 2024-03-29T11:33:54.293 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.294 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1347441480 0 0) 0x7f5528004eb0 con 0x7f553c0327d0 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f553c0c3780 con 0x7f553c0327d0 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 394733189 0 0) 0x7f553c0c39f0 con 0x7f553c02a640 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 >> v1:172.21.15.71:6790/0 conn(0x7f553c034850 legacy=0x7f553c165e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 >> v1:172.21.15.71:6789/0 conn(0x7f553c0327d0 legacy=0x7f553c0afd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 monclient: found mon.b 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0d5ab0 con 0x7f553c02a640 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient: _finish_auth 0 2024-03-29T11:33:54.295 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f5540ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f554923f640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f554923f640 10 monclient: _renew_subs 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f554923f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.291+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c0c3d30 con 0x7f553c02a640 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.292+0000 7f554923f640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.292+0000 7f554923f640 10 monclient: _renew_subs 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.292+0000 7f554923f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.292+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c0c2b90 con 0x7f553c02a640 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.292+0000 7f554923f640 1 librados: init done 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f554923f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f554923f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f554923f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c004cd0 con 0x7f553c02a640 2024-03-29T11:33:54.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5538001910 con 0x7f553c02a640 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5540ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5538001ce0 con 0x7f553c02a640 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5540ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5540ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5540ff9640 10 monclient: dump: 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:33:54.297 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:33:54.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.293+0000 7f5541ffb640 4 set_mon_vals no callback set 2024-03-29T11:33:54.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.297+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f553801aee0 con 0x7f553c02a640 2024-03-29T11:33:54.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.298+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f5538084ab0 con 0x7f553c02a640 2024-03-29T11:33:54.298 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.298+0000 7f5540ff9640 10 client.4300.objecter ms_dispatch 0x7f5544000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:54.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.298+0000 7f5540ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:54.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.298+0000 7f5540ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:54.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.298+0000 7f5540ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:33:54.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.299+0000 7f55437fe640 10 monclient: get_auth_request con 0x7f552806dfc0 auth_method 0 2024-03-29T11:33:54.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.299+0000 7f5540ff9640 10 client.4300.objecter ms_handle_connect 0x7f552806dfc0 2024-03-29T11:33:54.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.303+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5538050390 con 0x7f553c02a640 2024-03-29T11:33:54.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.303+0000 7f5540ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:54.301 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.303+0000 7f5540ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:54.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.634+0000 7f554923f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:54.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.635+0000 7f554923f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:54.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.635+0000 7f554923f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:33:54.633 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.635+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f553c0c3b80 con 0x7f553c02a640 2024-03-29T11:33:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.637+0000 7f5540ff9640 1 -- 172.21.15.71:0/3329914506 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f55380522c0 con 0x7f553c02a640 2024-03-29T11:33:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.637+0000 7f5540ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:54.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.637+0000 7f5540ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:54.635 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:33:54.636 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.641+0000 7f554923f640 10 librados: watch_flush enter 2024-03-29T11:33:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.641+0000 7f554923f640 10 librados: watch_flush exit 2024-03-29T11:33:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.641+0000 7f554923f640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:33:54.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.641+0000 7f554923f640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:33:54.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.643+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 >> v1:172.21.15.71:6816/1596514235 conn(0x7f552806dfc0 legacy=0x7f5528070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:54.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.643+0000 7f554923f640 10 monclient: shutdown 2024-03-29T11:33:54.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.643+0000 7f554923f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:54.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.643+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 >> v1:172.21.15.81:6789/0 conn(0x7f553c02a640 legacy=0x7f553c0c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.645+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 shutdown_connections 2024-03-29T11:33:54.642 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.645+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 >> 172.21.15.71:0/3329914506 conn(0x7f553c0058f0 msgr2=0x7f553c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.645+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 shutdown_connections 2024-03-29T11:33:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.645+0000 7f554923f640 1 -- 172.21.15.71:0/3329914506 wait complete. 2024-03-29T11:33:54.643 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:33:54.645+0000 7f554923f640 1 librados: shutdown 2024-03-29T11:33:54.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:01.468 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.628+0000 7fefb5b58640 10 monclient: get_monmap_and_config 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.628+0000 7fefb5b58640 10 monclient: build_initial_monmap 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.628+0000 7fefb5b58640 10 monclient: monmap: 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:01.629283+0000 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:01.629283+0000 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:01.625 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:01.626 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:01.626 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:01.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.629+0000 7fefb5b58640 1 Processor -- start 2024-03-29T11:34:01.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.629+0000 7fefb5b58640 1 -- start start 2024-03-29T11:34:01.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.629+0000 7fefb5b58640 10 monclient: init 2024-03-29T11:34:01.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.630+0000 7fefb5b58640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:01.627 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.630+0000 7fefb5b58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:01.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 10 monclient(hunting): picked mon.noname-a con 0x7fefb00b2800 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 10 monclient(hunting): picked mon.noname-c con 0x7fefb00a9c00 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 10 monclient(hunting): picked mon.noname-b con 0x7fefb00b38c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb0141eb0 con 0x7fefb00b2800 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb00b22d0 con 0x7fefb00a9c00 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb0142d30 con 0x7fefb00b38c0 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:01.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.631+0000 7fefb5b58640 10 monclient(hunting): authenticate will time out at 895.785642s 2024-03-29T11:34:01.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.635+0000 7fefb4b56640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fefb00b2800 0x7fefb0142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:44229/0 (socket says 172.21.15.71:44229) 2024-03-29T11:34:01.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.635+0000 7fefb4b56640 1 -- 172.21.15.71:0/2930130403 learned_addr learned my addr 172.21.15.71:0/2930130403 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774405615 0 0) 0x7fefb0141eb0 con 0x7fefb00b2800 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef98003520 con 0x7fefb00b2800 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317293582 0 0) 0x7fefb00b22d0 con 0x7fefb00a9c00 2024-03-29T11:34:01.634 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb0141eb0 con 0x7fefb00a9c00 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626854890 0 0) 0x7fefb0141eb0 con 0x7fefb00a9c00 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.636+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 >> v1:172.21.15.71:6790/0 conn(0x7fefb00b38c0 legacy=0x7fefb014d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 >> v1:172.21.15.71:6789/0 conn(0x7fefb00b2800 legacy=0x7fefb0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb00aaf90 con 0x7fefb00a9c00 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:01.635 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa0001670 con 0x7fefb00a9c00 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefaf7fe640 10 monclient: dump: 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:01.636 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.637 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.637 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefb5b58640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.637+0000 7fefb5b58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:01.638 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa00017d0 con 0x7fefb00a9c00 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefaf7fe640 1 -- 172.21.15.71:0/2930130403 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa0001ba0 con 0x7fefb00a9c00 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefaf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefaf7fe640 10 monclient: dump: 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:01.639 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefb5b58640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.641+0000 7fefb5b58640 4 set_mon_vals no callback set 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.642+0000 7fefb5b58640 10 monclient: shutdown 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.642+0000 7fefb5b58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.642+0000 7fefb5b58640 1 -- 172.21.15.71:0/2930130403 >> v1:172.21.15.81:6789/0 conn(0x7fefb00a9c00 legacy=0x7fefb013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.643+0000 7fefb5b58640 1 -- 172.21.15.71:0/2930130403 shutdown_connections 2024-03-29T11:34:01.640 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.643+0000 7fefb5b58640 1 -- 172.21.15.71:0/2930130403 >> 172.21.15.71:0/2930130403 conn(0x7fefb0066a00 msgr2=0x7fefb0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:01.641 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.645+0000 7fefb5b58640 1 -- 172.21.15.71:0/2930130403 shutdown_connections 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.646+0000 7fefb5b58640 1 -- 172.21.15.71:0/2930130403 wait complete. 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.647+0000 7fefb5b58640 10 monclient: build_initial_monmap 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.647+0000 7fefb5b58640 10 monclient: monmap: 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:01.644 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.647+0000 7fefb5b58640 1 librados: starting msgr at 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.647+0000 7fefb5b58640 1 librados: starting objecter 2024-03-29T11:34:01.645 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.647+0000 7fefb5b58640 1 Processor -- start 2024-03-29T11:34:01.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.649+0000 7fefb5b58640 1 -- start start 2024-03-29T11:34:01.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.649+0000 7fefb5b58640 1 librados: setting wanted keys 2024-03-29T11:34:01.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.649+0000 7fefb5b58640 1 librados: calling monclient init 2024-03-29T11:34:01.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.649+0000 7fefb5b58640 10 monclient: init 2024-03-29T11:34:01.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:01.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:01.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient(hunting): picked mon.noname-c con 0x7fefb00a9c00 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:01.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient(hunting): picked mon.noname-b con 0x7fefb00b2800 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:01.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient(hunting): picked mon.noname-a con 0x7fefb00b38c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:01.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb00ab760 con 0x7fefb00b38c0 2024-03-29T11:34:01.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb0071030 con 0x7fefb00b2800 2024-03-29T11:34:01.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb013afe0 con 0x7fefb00a9c00 2024-03-29T11:34:01.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:01.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.651+0000 7fefb5b58640 10 monclient(hunting): authenticate will time out at 895.805197s 2024-03-29T11:34:01.652 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.655+0000 7fefaffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fefb00b2800 0x7fefb006eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:52229/0 (socket says 172.21.15.71:52229) 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.655+0000 7fefaffff640 1 -- 172.21.15.71:0/306214860 learned_addr learned my addr 172.21.15.71:0/306214860 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.657+0000 7fefad7fa640 10 client.?.objecter ms_handle_connect 0x7fefb00b2800 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 client.?.objecter ms_handle_connect 0x7fefb00a9c00 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673865508 0 0) 0x7fefb013afe0 con 0x7fefb00a9c00 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:01.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:34:01.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.658+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef9c004eb0 con 0x7fefb00a9c00 2024-03-29T11:34:01.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.659+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769810345 0 0) 0x7fef9c004eb0 con 0x7fefb00a9c00 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.659+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefb013afe0 con 0x7fefb00a9c00 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.659+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa8001690 con 0x7fefb00a9c00 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.659+0000 7fefad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.659+0000 7fefad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.659+0000 7fefad7fa640 10 monclient(hunting): dump: 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:01.660 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 client.?.objecter ms_handle_connect 0x7fefb00b38c0 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1641015027 0 0) 0x7fefb013afe0 con 0x7fefb00a9c00 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 >> v1:172.21.15.81:6789/0 conn(0x7fefb00b2800 legacy=0x7fefb006eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 >> v1:172.21.15.71:6789/0 conn(0x7fefb00b38c0 legacy=0x7fefb013a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 1 monclient: found mon.c 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0072040 con 0x7fefb00a9c00 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:01.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 10 monclient: _renew_subs 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb0070c00 con 0x7fefb00a9c00 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 10 monclient: _renew_subs 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb0071580 con 0x7fefb00a9c00 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.660+0000 7fefb5b58640 1 librados: init done 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.661+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa8004b60 con 0x7fefb00a9c00 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.661+0000 7fefad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:01.662 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.662+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fefa8004e70 con 0x7fefb00a9c00 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.662+0000 7fefad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.662+0000 7fefad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.662+0000 7fefad7fa640 10 monclient: dump: 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.662+0000 7fefae7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:01.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.662+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fefa801ae40 con 0x7fefb00a9c00 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefb5b58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefb5b58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefb5b58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef90005080 con 0x7fefb00a9c00 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fefa8084d50 con 0x7fefb00a9c00 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefad7fa640 10 client.4313.objecter ms_dispatch 0x7fefb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefad7fa640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefad7fa640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:01.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefaffff640 10 monclient: get_auth_request con 0x7fef9c06dd70 auth_method 0 2024-03-29T11:34:01.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.663+0000 7fefad7fa640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T11:34:01.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.664+0000 7fefad7fa640 10 client.4313.objecter ms_handle_connect 0x7fef9c06dd70 2024-03-29T11:34:01.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.671+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefa8050630 con 0x7fefb00a9c00 2024-03-29T11:34:01.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.671+0000 7fefad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:01.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:01.671+0000 7fefad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:01.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.000+0000 7fefb5b58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:01.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.000+0000 7fefb5b58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:01.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.000+0000 7fefb5b58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:01.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.000+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef90005330 con 0x7fefb00a9c00 2024-03-29T11:34:01.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.001+0000 7fefad7fa640 1 -- 172.21.15.71:0/306214860 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fefa8052560 con 0x7fefb00a9c00 2024-03-29T11:34:01.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.002+0000 7fefad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:01.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.002+0000 7fefad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:01.999 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:01.999 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.005+0000 7fefb5b58640 10 librados: watch_flush enter 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.005+0000 7fefb5b58640 10 librados: watch_flush exit 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.005+0000 7fefb5b58640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.005+0000 7fefb5b58640 10 client.4313.objecter successfully canceled tick 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.005+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 >> v1:172.21.15.71:6816/1596514235 conn(0x7fef9c06dd70 legacy=0x7fef9c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 10 monclient: shutdown 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 >> v1:172.21.15.71:6790/0 conn(0x7fefb00a9c00 legacy=0x7fefb013a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 shutdown_connections 2024-03-29T11:34:02.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 >> 172.21.15.71:0/306214860 conn(0x7fefb0066a00 msgr2=0x7fefb00b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 shutdown_connections 2024-03-29T11:34:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 1 -- 172.21.15.71:0/306214860 wait complete. 2024-03-29T11:34:02.004 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:02.006+0000 7fefb5b58640 1 librados: shutdown 2024-03-29T11:34:02.019 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:08.826 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.994+0000 7f5f65606640 10 monclient: get_monmap_and_config 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.994+0000 7f5f65606640 10 monclient: build_initial_monmap 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.995+0000 7f5f65606640 10 monclient: monmap: 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:08.995940+0000 2024-03-29T11:34:08.992 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:08.995940+0000 2024-03-29T11:34:08.993 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:08.993 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:08.993 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:08.993 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:08.993 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:08.993 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:08.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.996+0000 7f5f65606640 1 Processor -- start 2024-03-29T11:34:08.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.996+0000 7f5f65606640 1 -- start start 2024-03-29T11:34:08.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.996+0000 7f5f65606640 10 monclient: init 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient(hunting): picked mon.noname-c con 0x7f5f600a9ac0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6013ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient(hunting): picked mon.noname-b con 0x7f5f600af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600afd10 con 0x7f5f6013ee90 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600af6f0 con 0x7f5f600a9ac0 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600aac80 con 0x7f5f600af900 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:08.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.997+0000 7f5f65606640 10 monclient(hunting): authenticate will time out at 903.151825s 2024-03-29T11:34:08.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.999+0000 7f5f5f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5f600af900 0x7f5f60149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48710/0 (socket says 172.21.15.71:48710) 2024-03-29T11:34:08.996 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:08.999+0000 7f5f5f7fe640 1 -- 172.21.15.71:0/1479631566 learned_addr learned my addr 172.21.15.71:0/1479631566 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630710885 0 0) 0x7f5f600aac80 con 0x7f5f600af900 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f48003520 con 0x7f5f600af900 2024-03-29T11:34:08.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3653943741 0 0) 0x7f5f48003520 con 0x7f5f600af900 2024-03-29T11:34:08.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.000+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 >> v1:172.21.15.81:6789/0 conn(0x7f5f600a9ac0 legacy=0x7f5f600adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:08.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 >> v1:172.21.15.71:6789/0 conn(0x7f5f6013ee90 legacy=0x7f5f6013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:08.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:08.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 1 monclient: found mon.noname-b 2024-03-29T11:34:08.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:08.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600aaf60 con 0x7f5f600af900 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f65606640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f65606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5f50001670 con 0x7f5f600af900 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.002+0000 7f5f5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.003+0000 7f5f5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.003+0000 7f5f5dffb640 10 monclient: dump: 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.000 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:09.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.003+0000 7f5f65606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f500018d0 con 0x7f5f600af900 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f5dffb640 1 -- 172.21.15.71:0/1479631566 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5f50001d20 con 0x7f5f600af900 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f5dffb640 10 monclient: dump: 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.002 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f65606640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.004+0000 7f5f65606640 4 set_mon_vals no callback set 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.005+0000 7f5f65606640 10 monclient: shutdown 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.005+0000 7f5f65606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.005+0000 7f5f65606640 1 -- 172.21.15.71:0/1479631566 >> v1:172.21.15.71:6790/0 conn(0x7f5f600af900 legacy=0x7f5f60149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.005+0000 7f5f65606640 1 -- 172.21.15.71:0/1479631566 shutdown_connections 2024-03-29T11:34:09.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.005+0000 7f5f65606640 1 -- 172.21.15.71:0/1479631566 >> 172.21.15.71:0/1479631566 conn(0x7f5f600b35e0 msgr2=0x7f5f600b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:09.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.008+0000 7f5f65606640 1 -- 172.21.15.71:0/1479631566 shutdown_connections 2024-03-29T11:34:09.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.008+0000 7f5f65606640 1 -- 172.21.15.71:0/1479631566 wait complete. 2024-03-29T11:34:09.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.009+0000 7f5f65606640 10 monclient: build_initial_monmap 2024-03-29T11:34:09.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.010+0000 7f5f65606640 10 monclient: monmap: 2024-03-29T11:34:09.006 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:09.007 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.010+0000 7f5f65606640 1 librados: starting msgr at 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.010+0000 7f5f65606640 1 librados: starting objecter 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.011+0000 7f5f65606640 1 Processor -- start 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.011+0000 7f5f65606640 1 -- start start 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.011+0000 7f5f65606640 1 librados: setting wanted keys 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.011+0000 7f5f65606640 1 librados: calling monclient init 2024-03-29T11:34:09.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.011+0000 7f5f65606640 10 monclient: init 2024-03-29T11:34:09.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.012+0000 7f5f65606640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.013+0000 7f5f65606640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.013+0000 7f5f65606640 10 monclient(hunting): picked mon.noname-c con 0x7f5f600a9ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.013+0000 7f5f5effd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5f600a9ac0 0x7f5f60070890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49300/0 (socket says 172.21.15.71:49300) 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 10 monclient(hunting): picked mon.noname-b con 0x7f5f600af900 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6013ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f60136eb0 con 0x7f5f6013ee90 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f60135c50 con 0x7f5f600af900 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f601370e0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:09.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.014+0000 7f5f65606640 10 monclient(hunting): authenticate will time out at 903.169033s 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f5effd640 1 -- 172.21.15.71:0/4177332707 learned_addr learned my addr 172.21.15.71:0/4177332707 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 10 client.?.objecter ms_handle_connect 0x7f5f600a9ac0 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233428743 0 0) 0x7f5f601370e0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.015+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c004eb0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1898352104 0 0) 0x7f5f4c004eb0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f601370e0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5f54001870 con 0x7f5f600a9ac0 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:09.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 10 monclient(hunting): dump: 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:09.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.016+0000 7f5f477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:09.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 client.?.objecter ms_handle_connect 0x7f5f600af900 2024-03-29T11:34:09.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:09.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 client.?.objecter ms_handle_connect 0x7f5f6013ee90 2024-03-29T11:34:09.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:09.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396536641 0 0) 0x7f5f60135c50 con 0x7f5f600af900 2024-03-29T11:34:09.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c004eb0 con 0x7f5f600af900 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061695977 0 0) 0x7f5f60136eb0 con 0x7f5f6013ee90 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60135c50 con 0x7f5f6013ee90 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1433471374 0 0) 0x7f5f601370e0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 >> v1:172.21.15.81:6789/0 conn(0x7f5f600af900 legacy=0x7f5f601347a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 >> v1:172.21.15.71:6789/0 conn(0x7f5f6013ee90 legacy=0x7f5f601da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 monclient: found mon.c 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f601373c0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.018+0000 7f5f65606640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 10 monclient: _renew_subs 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f601376a0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:09.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 10 monclient: _renew_subs 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f601358e0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f65606640 1 librados: init done 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f540019d0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5f54004aa0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f477fe640 10 monclient: dump: 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:09.020 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.019+0000 7f5f5d7fa640 4 set_mon_vals no callback set 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.022+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f5f5401aaf0 con 0x7f5f600a9ac0 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.023+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f5f54084b70 con 0x7f5f600a9ac0 2024-03-29T11:34:09.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.023+0000 7f5f477fe640 10 client.4319.objecter ms_dispatch 0x7f5f60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:09.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.023+0000 7f5f477fe640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:09.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.023+0000 7f5f477fe640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:09.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.023+0000 7f5f477fe640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:34:09.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.024+0000 7f5f5e7fc640 10 monclient: get_auth_request con 0x7f5f4c06de00 auth_method 0 2024-03-29T11:34:09.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.024+0000 7f5f65606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:09.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.025+0000 7f5f477fe640 10 client.4319.objecter ms_handle_connect 0x7f5f4c06de00 2024-03-29T11:34:09.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.025+0000 7f5f65606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:09.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.025+0000 7f5f65606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:09.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.025+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40005080 con 0x7f5f600a9ac0 2024-03-29T11:34:09.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.035+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f54050280 con 0x7f5f600a9ac0 2024-03-29T11:34:09.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.035+0000 7f5f477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:09.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.035+0000 7f5f477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:09.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.394+0000 7f5f65606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:09.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.394+0000 7f5f65606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:09.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.394+0000 7f5f65606640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:09.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.394+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f40005330 con 0x7f5f600a9ac0 2024-03-29T11:34:09.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.395+0000 7f5f477fe640 1 -- 172.21.15.71:0/4177332707 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f5f54052380 con 0x7f5f600a9ac0 2024-03-29T11:34:09.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.395+0000 7f5f477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:09.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.395+0000 7f5f477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:09.392 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:09.393 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 10 librados: watch_flush enter 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 10 librados: watch_flush exit 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 >> v1:172.21.15.71:6816/1596514235 conn(0x7f5f4c06de00 legacy=0x7f5f4c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 10 monclient: shutdown 2024-03-29T11:34:09.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:09.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 >> v1:172.21.15.71:6790/0 conn(0x7f5f600a9ac0 legacy=0x7f5f60070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:09.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 shutdown_connections 2024-03-29T11:34:09.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.399+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 >> 172.21.15.71:0/4177332707 conn(0x7f5f600b35e0 msgr2=0x7f5f600ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:09.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.400+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 shutdown_connections 2024-03-29T11:34:09.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.400+0000 7f5f65606640 1 -- 172.21.15.71:0/4177332707 wait complete. 2024-03-29T11:34:09.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:09.400+0000 7f5f65606640 1 librados: shutdown 2024-03-29T11:34:09.412 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:16.219 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:16.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.371+0000 7f3a32872640 10 monclient: get_monmap_and_config 2024-03-29T11:34:16.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.371+0000 7f3a32872640 10 monclient: build_initial_monmap 2024-03-29T11:34:16.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.371+0000 7f3a32872640 10 monclient: monmap: 2024-03-29T11:34:16.368 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:16.368 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:16.372350+0000 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:16.372350+0000 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:16.369 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.372+0000 7f3a32872640 1 Processor -- start 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.372+0000 7f3a32872640 1 -- start start 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.372+0000 7f3a32872640 10 monclient: init 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c071c30 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:16.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c0a9750 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:16.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c13e1e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:16.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0ab300 con 0x7f3a2c071c30 2024-03-29T11:34:16.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0ac2f0 con 0x7f3a2c0a9750 2024-03-29T11:34:16.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0ad8d0 con 0x7f3a2c13e1e0 2024-03-29T11:34:16.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:16.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a32872640 10 monclient(hunting): authenticate will time out at 910.527656s 2024-03-29T11:34:16.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a2bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f3a2c071c30 0x7f3a2c0b0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:56190/0 (socket says 172.21.15.71:56190) 2024-03-29T11:34:16.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a2bfff640 1 -- 172.21.15.71:0/3524184125 learned_addr learned my addr 172.21.15.71:0/3524184125 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.373+0000 7f3a30e0f640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3a2c13e1e0 0x7f3a2c0ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46710/0 (socket says 172.21.15.71:46710) 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683439734 0 0) 0x7f3a2c0ab300 con 0x7f3a2c071c30 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a14003520 con 0x7f3a2c071c30 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819750710 0 0) 0x7f3a2c0ac2f0 con 0x7f3a2c0a9750 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:16.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.374+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c0ab300 con 0x7f3a2c0a9750 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2114091842 0 0) 0x7f3a14003520 con 0x7f3a2c071c30 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 >> v1:172.21.15.71:6790/0 conn(0x7f3a2c13e1e0 legacy=0x7f3a2c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 >> v1:172.21.15.81:6789/0 conn(0x7f3a2c0a9750 legacy=0x7f3a2c13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 1 monclient: found mon.noname-a 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c0ada60 con 0x7f3a2c071c30 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: _finish_auth 0 2024-03-29T11:34:16.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a32872640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a32872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3a24001a10 con 0x7f3a2c071c30 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a2affd640 10 monclient: dump: 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.384 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.375+0000 7f3a32872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a24001b70 con 0x7f3a2c071c30 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a32872640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a32872640 4 set_mon_vals no callback set 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a2affd640 1 -- 172.21.15.71:0/3524184125 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3a24001fc0 con 0x7f3a2c071c30 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:16.385 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a2affd640 10 monclient: dump: 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a32872640 10 monclient: shutdown 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a32872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.376+0000 7f3a32872640 1 -- 172.21.15.71:0/3524184125 >> v1:172.21.15.71:6789/0 conn(0x7f3a2c071c30 legacy=0x7f3a2c0b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.377+0000 7f3a32872640 1 -- 172.21.15.71:0/3524184125 shutdown_connections 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.377+0000 7f3a32872640 1 -- 172.21.15.71:0/3524184125 >> 172.21.15.71:0/3524184125 conn(0x7f3a2c06b660 msgr2=0x7f3a2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.377+0000 7f3a32872640 1 -- 172.21.15.71:0/3524184125 shutdown_connections 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.377+0000 7f3a32872640 1 -- 172.21.15.71:0/3524184125 wait complete. 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.378+0000 7f3a32872640 10 monclient: build_initial_monmap 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.378+0000 7f3a32872640 10 monclient: monmap: 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:16.387 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.378+0000 7f3a32872640 1 librados: starting msgr at 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.378+0000 7f3a32872640 1 librados: starting objecter 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.379+0000 7f3a32872640 1 Processor -- start 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.379+0000 7f3a32872640 1 -- start start 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.379+0000 7f3a32872640 1 librados: setting wanted keys 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.379+0000 7f3a32872640 1 librados: calling monclient init 2024-03-29T11:34:16.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.379+0000 7f3a32872640 10 monclient: init 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c071c30 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c0a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c13e1e0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c142330 con 0x7f3a2c0a9750 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c141130 con 0x7f3a2c13e1e0 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c1412c0 con 0x7f3a2c071c30 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a32872640 10 monclient(hunting): authenticate will time out at 910.534550s 2024-03-29T11:34:16.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a2bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3a2c071c30 0x7f3a2c141b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47286/0 (socket says 172.21.15.71:47286) 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a2bfff640 1 -- 172.21.15.71:0/4200497344 learned_addr learned my addr 172.21.15.71:0/4200497344 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a28ff9640 10 client.?.objecter ms_handle_connect 0x7f3a2c071c30 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.380+0000 7f3a28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 client.?.objecter ms_handle_connect 0x7f3a2c0a9750 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236385582 0 0) 0x7f3a2c142330 con 0x7f3a2c0a9750 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a18004eb0 con 0x7f3a2c0a9750 2024-03-29T11:34:16.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232764101 0 0) 0x7f3a2c1412c0 con 0x7f3a2c071c30 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c142330 con 0x7f3a2c071c30 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3555970230 0 0) 0x7f3a18004eb0 con 0x7f3a2c0a9750 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a2c1412c0 con 0x7f3a2c0a9750 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3a20001a50 con 0x7f3a2c0a9750 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:16.391 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 295897424 0 0) 0x7f3a2c142330 con 0x7f3a2c071c30 2024-03-29T11:34:16.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a18004eb0 con 0x7f3a2c071c30 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3a24001f80 con 0x7f3a2c071c30 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.381+0000 7f3a28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.393 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 client.?.objecter ms_handle_connect 0x7f3a2c13e1e0 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2793712805 0 0) 0x7f3a18004eb0 con 0x7f3a2c071c30 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 >> v1:172.21.15.81:6789/0 conn(0x7f3a2c13e1e0 legacy=0x7f3a2c1db8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 >> v1:172.21.15.71:6789/0 conn(0x7f3a2c0a9750 legacy=0x7f3a2c070d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 1 monclient: found mon.c 2024-03-29T11:34:16.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c141450 con 0x7f3a2c071c30 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 10 monclient: _renew_subs 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c0b2720 con 0x7f3a2c071c30 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 10 monclient: _renew_subs 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:16.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c071460 con 0x7f3a2c071c30 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.382+0000 7f3a32872640 1 librados: init done 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a24004e10 con 0x7f3a2c071c30 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3a24004f80 con 0x7f3a2c071c30 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a28ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a28ff9640 10 monclient: dump: 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:16.396 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a2a7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.383+0000 7f3a32872640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.384+0000 7f3a32872640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.384+0000 7f3a32872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:16.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.384+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c005080 con 0x7f3a2c071c30 2024-03-29T11:34:16.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.385+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f3a2401af50 con 0x7f3a2c071c30 2024-03-29T11:34:16.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f3a24084c90 con 0x7f3a2c071c30 2024-03-29T11:34:16.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a28ff9640 10 client.4325.objecter ms_dispatch 0x7f3a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a28ff9640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a28ff9640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a28ff9640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a2b7fe640 10 monclient: get_auth_request con 0x7f3a1806dec0 auth_method 0 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.386+0000 7f3a28ff9640 10 client.4325.objecter ms_handle_connect 0x7f3a1806dec0 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.397+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a240505d0 con 0x7f3a2c071c30 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.397+0000 7f3a28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:16.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.397+0000 7f3a28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:16.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.772+0000 7f3a32872640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:16.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.772+0000 7f3a32872640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:16.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.772+0000 7f3a32872640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:16.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.772+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a0c005330 con 0x7f3a2c071c30 2024-03-29T11:34:16.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.773+0000 7f3a28ff9640 1 -- 172.21.15.71:0/4200497344 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f3a240505d0 con 0x7f3a2c071c30 2024-03-29T11:34:16.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.773+0000 7f3a28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:16.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.773+0000 7f3a28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:16.771 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:16.771 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:16.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 10 librados: watch_flush enter 2024-03-29T11:34:16.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 10 librados: watch_flush exit 2024-03-29T11:34:16.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:34:16.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:34:16.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 >> v1:172.21.15.71:6816/1596514235 conn(0x7f3a1806dec0 legacy=0x7f3a18070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 10 monclient: shutdown 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 >> v1:172.21.15.71:6790/0 conn(0x7f3a2c071c30 legacy=0x7f3a2c141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 shutdown_connections 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.777+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 >> 172.21.15.71:0/4200497344 conn(0x7f3a2c06b660 msgr2=0x7f3a2c139c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.778+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 shutdown_connections 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.778+0000 7f3a32872640 1 -- 172.21.15.71:0/4200497344 wait complete. 2024-03-29T11:34:16.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:16.778+0000 7f3a32872640 1 librados: shutdown 2024-03-29T11:34:16.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:23.596 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:23.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.729+0000 7f208b6cf640 10 monclient: get_monmap_and_config 2024-03-29T11:34:23.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.729+0000 7f208b6cf640 10 monclient: build_initial_monmap 2024-03-29T11:34:23.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.730+0000 7f208b6cf640 10 monclient: monmap: 2024-03-29T11:34:23.727 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:23.727 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:23.727 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:23.730932+0000 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:23.730932+0000 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:23.728 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:23.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.731+0000 7f208b6cf640 1 Processor -- start 2024-03-29T11:34:23.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 1 -- start start 2024-03-29T11:34:23.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 10 monclient: init 2024-03-29T11:34:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f20841457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:23.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f208413f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f20840a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.732+0000 7f208b6cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20840aaef0 con 0x7f20840a9d70 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f208b6cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2084145bd0 con 0x7f20841457f0 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f208b6cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2084145eb0 con 0x7f208413f650 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f208b6cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f2088c6a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f208413f650 0x7f20840b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53892/0 (socket says 172.21.15.71:53892) 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f2088c6a640 1 -- 172.21.15.71:0/2716136013 learned_addr learned my addr 172.21.15.71:0/2716136013 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f208b6cf640 10 monclient(hunting): authenticate will time out at 917.887265s 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294771678 0 0) 0x7f2084145eb0 con 0x7f208413f650 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:34:23.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2070003520 con 0x7f208413f650 2024-03-29T11:34:23.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 230850278 0 0) 0x7f2070003520 con 0x7f208413f650 2024-03-29T11:34:23.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 >> v1:172.21.15.81:6789/0 conn(0x7f20841457f0 legacy=0x7f208413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.733+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 >> v1:172.21.15.71:6789/0 conn(0x7f20840a9d70 legacy=0x7f208414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 1 monclient: found mon.noname-b 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084146340 con 0x7f208413f650 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: _finish_auth 0 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:23.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2074001690 con 0x7f208413f650 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: dump: 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:23.736 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f208b6cf640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f208b6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2074001920 con 0x7f208413f650 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f207bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:23.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f208b6cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.735+0000 7f208b6cf640 4 set_mon_vals no callback set 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f207bfff640 1 -- 172.21.15.71:0/2716136013 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2074001d30 con 0x7f208413f650 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f207bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f207bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f207bfff640 10 monclient: dump: 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.738 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f208b6cf640 10 monclient: shutdown 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f208b6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f208b6cf640 1 -- 172.21.15.71:0/2716136013 >> v1:172.21.15.71:6790/0 conn(0x7f208413f650 legacy=0x7f20840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f208b6cf640 1 -- 172.21.15.71:0/2716136013 shutdown_connections 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.736+0000 7f208b6cf640 1 -- 172.21.15.71:0/2716136013 >> 172.21.15.71:0/2716136013 conn(0x7f2084066a00 msgr2=0x7f2084066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.737+0000 7f208b6cf640 1 -- 172.21.15.71:0/2716136013 shutdown_connections 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.737+0000 7f208b6cf640 1 -- 172.21.15.71:0/2716136013 wait complete. 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.737+0000 7f208b6cf640 10 monclient: build_initial_monmap 2024-03-29T11:34:23.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.738+0000 7f208b6cf640 10 monclient: monmap: 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:23.740 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.738+0000 7f208b6cf640 1 librados: starting msgr at 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.738+0000 7f208b6cf640 1 librados: starting objecter 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.738+0000 7f208b6cf640 1 Processor -- start 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.739+0000 7f208b6cf640 1 -- start start 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.739+0000 7f208b6cf640 1 librados: setting wanted keys 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.739+0000 7f208b6cf640 1 librados: calling monclient init 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.739+0000 7f208b6cf640 10 monclient: init 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.740+0000 7f208b6cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.740+0000 7f208b6cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:23.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.740+0000 7f208b6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f20840a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.740+0000 7f208b6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f20841457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208946b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f20840a9d70 0x7f20841d7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57380/0 (socket says 172.21.15.71:57380) 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208b6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f20841429a0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208b6cf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20841d78e0 con 0x7f20841457f0 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208b6cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20841d8480 con 0x7f20841429a0 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208b6cf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2084141cc0 con 0x7f20840a9d70 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208b6cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208b6cf640 10 monclient(hunting): authenticate will time out at 917.895752s 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.741+0000 7f208946b640 1 -- 172.21.15.71:0/1540148030 learned_addr learned my addr 172.21.15.71:0/1540148030 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.742+0000 7f2079ffb640 10 client.?.objecter ms_handle_connect 0x7f20840a9d70 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.742+0000 7f2079ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134412556 0 0) 0x7f2084141cc0 con 0x7f20840a9d70 2024-03-29T11:34:23.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2068004eb0 con 0x7f20840a9d70 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1411779422 0 0) 0x7f2068004eb0 con 0x7f20840a9d70 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2084141cc0 con 0x7f20840a9d70 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2080001950 con 0x7f20840a9d70 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.751 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.743+0000 7f2079ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.744+0000 7f2079ffb640 10 client.?.objecter ms_handle_connect 0x7f20841429a0 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.744+0000 7f2079ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.744+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1685562597 0 0) 0x7f2084141cc0 con 0x7f20840a9d70 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.744+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 >> v1:172.21.15.81:6789/0 conn(0x7f20841429a0 legacy=0x7f20841404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.744+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 >> v1:172.21.15.71:6789/0 conn(0x7f20841457f0 legacy=0x7f208413fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.744+0000 7f2079ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 1 monclient: found mon.c 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:23.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084141f30 con 0x7f20840a9d70 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f208b6cf640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f208b6cf640 10 monclient: _renew_subs 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f208b6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20841d7570 con 0x7f20840a9d70 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f208b6cf640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.745+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2080001ab0 con 0x7f20840a9d70 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f2079ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2080003a60 con 0x7f20840a9d70 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f2079ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f2079ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:23.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f2079ffb640 10 monclient: dump: 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f208b6cf640 10 monclient: _renew_subs 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f208b6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f208413bb80 con 0x7f20840a9d70 2024-03-29T11:34:23.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f208b6cf640 1 librados: init done 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.746+0000 7f207b7fe640 4 set_mon_vals no callback set 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.747+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f2080003bc0 con 0x7f20840a9d70 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f208b6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f208b6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f208b6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2064005080 con 0x7f20840a9d70 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f2080084b00 con 0x7f20840a9d70 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f2088c6a640 10 monclient: get_auth_request con 0x7f206806dd30 auth_method 0 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f2079ffb640 10 client.4331.objecter ms_dispatch 0x7f2084000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.748+0000 7f2079ffb640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.749+0000 7f2079ffb640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.749+0000 7f2079ffb640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:34:23.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.749+0000 7f2079ffb640 10 client.4331.objecter ms_handle_connect 0x7f206806dd30 2024-03-29T11:34:23.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.755+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20800503e0 con 0x7f20840a9d70 2024-03-29T11:34:23.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.755+0000 7f2079ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:23.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:23.755+0000 7f2079ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:24.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.064+0000 7f208b6cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:24.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.064+0000 7f208b6cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:24.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.064+0000 7f208b6cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:24.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.064+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2064005330 con 0x7f20840a9d70 2024-03-29T11:34:24.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.065+0000 7f2079ffb640 1 -- 172.21.15.71:0/1540148030 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f2080052310 con 0x7f20840a9d70 2024-03-29T11:34:24.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.065+0000 7f2079ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:24.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.065+0000 7f2079ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:24.063 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:24.063 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.067+0000 7f208b6cf640 10 librados: watch_flush enter 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.067+0000 7f208b6cf640 10 librados: watch_flush exit 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.067+0000 7f208b6cf640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.067+0000 7f208b6cf640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 >> v1:172.21.15.71:6816/1596514235 conn(0x7f206806dd30 legacy=0x7f20680701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 10 monclient: shutdown 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 >> v1:172.21.15.71:6790/0 conn(0x7f20840a9d70 legacy=0x7f20841d7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 shutdown_connections 2024-03-29T11:34:24.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 >> 172.21.15.71:0/1540148030 conn(0x7f2084066a00 msgr2=0x7f20840b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:24.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 shutdown_connections 2024-03-29T11:34:24.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 -- 172.21.15.71:0/1540148030 wait complete. 2024-03-29T11:34:24.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:24.068+0000 7f208b6cf640 1 librados: shutdown 2024-03-29T11:34:24.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:30.888 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:31.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.040+0000 7f8dc66d0640 10 monclient: get_monmap_and_config 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.040+0000 7f8dc66d0640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.040+0000 7f8dc66d0640 10 monclient: monmap: 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:31.041061+0000 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:31.041061+0000 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:31.038 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:31.039 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:31.039 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:31.039 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:31.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.041+0000 7f8dc66d0640 1 Processor -- start 2024-03-29T11:34:31.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.041+0000 7f8dc66d0640 1 -- start start 2024-03-29T11:34:31.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.041+0000 7f8dc66d0640 10 monclient: init 2024-03-29T11:34:31.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:31.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient(hunting): picked mon.noname-b con 0x7f8db8034850 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:31.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient(hunting): picked mon.noname-a con 0x7f8db80c4ec0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:31.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient(hunting): picked mon.noname-c con 0x7f8db80c7d30 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:31.043 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db80c83c0 con 0x7f8db80c4ec0 2024-03-29T11:34:31.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db8033280 con 0x7f8db80c7d30 2024-03-29T11:34:31.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db8029100 con 0x7f8db8034850 2024-03-29T11:34:31.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.042+0000 7f8dc66d0640 10 monclient(hunting): authenticate will time out at 925.196538s 2024-03-29T11:34:31.044 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.046+0000 7f8dc56ce640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8db8034850 0x7f8db80c47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56318/0 (socket says 172.21.15.71:56318) 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.046+0000 7f8dc56ce640 1 -- 172.21.15.71:0/551006074 learned_addr learned my addr 172.21.15.71:0/551006074 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104595048 0 0) 0x7f8db8029100 con 0x7f8db8034850 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dac003520 con 0x7f8db8034850 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230816388 0 0) 0x7f8dac003520 con 0x7f8db8034850 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 >> v1:172.21.15.81:6789/0 conn(0x7f8db80c7d30 legacy=0x7f8db8029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:31.045 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.047+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 >> v1:172.21.15.71:6789/0 conn(0x7f8db80c4ec0 legacy=0x7f8db8029800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 1 monclient: found mon.noname-b 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80293e0 con 0x7f8db8034850 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8dbc001670 con 0x7f8db8034850 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: dump: 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:31.048 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8dc66d0640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8dc66d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dbc0018d0 con 0x7f8db8034850 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8db7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8dc66d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:31.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.050+0000 7f8dc66d0640 4 set_mon_vals no callback set 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.051+0000 7f8dc66d0640 10 monclient: shutdown 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.051+0000 7f8dc66d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.051+0000 7f8dc66d0640 1 -- 172.21.15.71:0/551006074 >> v1:172.21.15.71:6790/0 conn(0x7f8db8034850 legacy=0x7f8db80c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.051+0000 7f8db7fff640 1 -- 172.21.15.71:0/551006074 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8dbc001ca0 con 0x7f8db8034850 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.051+0000 7f8db7fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.052+0000 7f8dc66d0640 1 -- 172.21.15.71:0/551006074 shutdown_connections 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.052+0000 7f8dc66d0640 1 -- 172.21.15.71:0/551006074 >> 172.21.15.71:0/551006074 conn(0x7f8db80058f0 msgr2=0x7f8db8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.052+0000 7f8dc66d0640 1 -- 172.21.15.71:0/551006074 shutdown_connections 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.052+0000 7f8dc66d0640 1 -- 172.21.15.71:0/551006074 wait complete. 2024-03-29T11:34:31.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 10 monclient: build_initial_monmap 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 10 monclient: monmap: 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 1 librados: starting msgr at 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 1 librados: starting objecter 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 1 Processor -- start 2024-03-29T11:34:31.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 1 -- start start 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 1 librados: setting wanted keys 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 1 librados: calling monclient init 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.053+0000 7f8dc66d0640 10 monclient: init 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient(hunting): picked mon.noname-a con 0x7f8db8034850 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient(hunting): picked mon.noname-b con 0x7f8db80c4ec0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient(hunting): picked mon.noname-c con 0x7f8db80c7d30 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db815fbd0 con 0x7f8db8034850 2024-03-29T11:34:31.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db80cab70 con 0x7f8db80c4ec0 2024-03-29T11:34:31.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8db8160a50 con 0x7f8db80c7d30 2024-03-29T11:34:31.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:31.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.054+0000 7f8dc66d0640 10 monclient(hunting): authenticate will time out at 925.208825s 2024-03-29T11:34:31.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8dc5ecf640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8db80c7d30 0x7f8db8167170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57850/0 (socket says 172.21.15.71:57850) 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8dc5ecf640 1 -- 172.21.15.71:0/1068836565 learned_addr learned my addr 172.21.15.71:0/1068836565 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 client.?.objecter ms_handle_connect 0x7f8db80c7d30 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 client.?.objecter ms_handle_connect 0x7f8db80c4ec0 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74087409 0 0) 0x7f8db8160a50 con 0x7f8db80c7d30 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8da4004eb0 con 0x7f8db80c7d30 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706000603 0 0) 0x7f8db80cab70 con 0x7f8db80c4ec0 2024-03-29T11:34:31.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.057+0000 7f8db5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8db8160a50 con 0x7f8db80c4ec0 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2518539905 0 0) 0x7f8da4004eb0 con 0x7f8db80c7d30 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8db80cab70 con 0x7f8db80c7d30 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8db0003f60 con 0x7f8db80c7d30 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:31.061 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126299413 0 0) 0x7f8db8160a50 con 0x7f8db80c4ec0 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8da4004eb0 con 0x7f8db80c4ec0 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8dc004fdd0 con 0x7f8db80c4ec0 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:31.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient(hunting): dump: 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3583713013 0 0) 0x7f8db80cab70 con 0x7f8db80c7d30 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 >> v1:172.21.15.81:6789/0 conn(0x7f8db80c4ec0 legacy=0x7f8db8160340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 >> v1:172.21.15.71:6789/0 conn(0x7f8db8034850 legacy=0x7f8db80ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 monclient: found mon.c 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db80cbab0 con 0x7f8db80c7d30 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8dc66d0640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8dc66d0640 10 monclient: _renew_subs 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8dc66d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.058+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db80c9730 con 0x7f8db80c7d30 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8dc66d0640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8dc66d0640 10 monclient: _renew_subs 2024-03-29T11:34:31.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8dc66d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db80c6040 con 0x7f8db80c7d30 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8dc66d0640 1 librados: init done 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8db0004130 con 0x7f8db80c7d30 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8db5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.059+0000 7f8db77fe640 4 set_mon_vals no callback set 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.061+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8db0001070 con 0x7f8db80c7d30 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.061+0000 7f8db5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.061+0000 7f8db5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.061+0000 7f8db5ffb640 10 monclient: dump: 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:31.065 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.061+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8db00012b0 con 0x7f8db80c7d30 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8dc56ce640 10 monclient: get_auth_request con 0x7f8da406dcd0 auth_method 0 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f8db00849a0 con 0x7f8db80c7d30 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8db5ffb640 10 client.4337.objecter ms_dispatch 0x7f8dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8db5ffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8db5ffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8db5ffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:34:31.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.062+0000 7f8db5ffb640 10 client.4337.objecter ms_handle_connect 0x7f8da406dcd0 2024-03-29T11:34:31.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.063+0000 7f8dc66d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.063+0000 7f8dc66d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.063+0000 7f8dc66d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:31.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.063+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da0005080 con 0x7f8db80c7d30 2024-03-29T11:34:31.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.076+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8db0050360 con 0x7f8db80c7d30 2024-03-29T11:34:31.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.076+0000 7f8db5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:31.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.076+0000 7f8db5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8dc66d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8dc66d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8dc66d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8da0005330 con 0x7f8db80c7d30 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8db5ffb640 1 -- 172.21.15.71:0/1068836565 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f8db0052290 con 0x7f8db80c7d30 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8db5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:31.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.410+0000 7f8db5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:31.409 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:31.409 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:31.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.414+0000 7f8dc66d0640 10 librados: watch_flush enter 2024-03-29T11:34:31.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 10 librados: watch_flush exit 2024-03-29T11:34:31.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:34:31.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 >> v1:172.21.15.71:6816/1596514235 conn(0x7f8da406dcd0 legacy=0x7f8da4070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 10 monclient: shutdown 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 >> v1:172.21.15.71:6790/0 conn(0x7f8db80c7d30 legacy=0x7f8db8167170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 shutdown_connections 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.415+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 >> 172.21.15.71:0/1068836565 conn(0x7f8db80058f0 msgr2=0x7f8db80c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:31.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.416+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 shutdown_connections 2024-03-29T11:34:31.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.416+0000 7f8dc66d0640 1 -- 172.21.15.71:0/1068836565 wait complete. 2024-03-29T11:34:31.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:31.416+0000 7f8dc66d0640 1 librados: shutdown 2024-03-29T11:34:31.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:38.237 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.387+0000 7f13e498c640 10 monclient: get_monmap_and_config 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.387+0000 7f13e498c640 10 monclient: build_initial_monmap 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.387+0000 7f13e498c640 10 monclient: monmap: 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:38.388834+0000 2024-03-29T11:34:38.386 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:38.388834+0000 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.390+0000 7f13e498c640 1 Processor -- start 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.390+0000 7f13e498c640 1 -- start start 2024-03-29T11:34:38.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.390+0000 7f13e498c640 10 monclient: init 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient(hunting): picked mon.noname-b con 0x7f13e00a9930 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient(hunting): picked mon.noname-a con 0x7f13e0149dc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient(hunting): picked mon.noname-c con 0x7f13e014ff40 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00aae60 con 0x7f13e0149dc0 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b4310 con 0x7f13e014ff40 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00afdd0 con 0x7f13e00a9930 2024-03-29T11:34:38.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:38.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.391+0000 7f13e498c640 10 monclient(hunting): authenticate will time out at 932.545668s 2024-03-29T11:34:38.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.394+0000 7f13ded9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f13e014ff40 0x7f13e0152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:56691/0 (socket says 172.21.15.71:56691) 2024-03-29T11:34:38.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.394+0000 7f13ded9d640 1 -- 172.21.15.71:0/2154287545 learned_addr learned my addr 172.21.15.71:0/2154287545 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:38.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.397+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931646905 0 0) 0x7f13e00b4310 con 0x7f13e014ff40 2024-03-29T11:34:38.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.397+0000 7f13dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.397+0000 7f13dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.397+0000 7f13dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.397+0000 7f13dd59a640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:34:38.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.397+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c8003520 con 0x7f13e014ff40 2024-03-29T11:34:38.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.400+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591106638 0 0) 0x7f13c8003520 con 0x7f13e014ff40 2024-03-29T11:34:38.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 >> v1:172.21.15.71:6790/0 conn(0x7f13e00a9930 legacy=0x7f13e0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:38.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 >> v1:172.21.15.71:6789/0 conn(0x7f13e0149dc0 legacy=0x7f13e014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:38.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:38.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 1 monclient: found mon.noname-c 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e014a950 con 0x7f13e014ff40 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: _finish_auth 0 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f13d8001a10 con 0x7f13e014ff40 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13dd59a640 10 monclient: dump: 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:38.399 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13e498c640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:34:38.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.401+0000 7f13e498c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.402+0000 7f13dd59a640 1 -- 172.21.15.71:0/2154287545 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d8001be0 con 0x7f13e014ff40 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.402+0000 7f13dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.402+0000 7f13e498c640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.402+0000 7f13e498c640 4 set_mon_vals no callback set 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.403+0000 7f13e498c640 10 monclient: shutdown 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.403+0000 7f13e498c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:38.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.403+0000 7f13e498c640 1 -- 172.21.15.71:0/2154287545 >> v1:172.21.15.81:6789/0 conn(0x7f13e014ff40 legacy=0x7f13e0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.403+0000 7f13e498c640 1 -- 172.21.15.71:0/2154287545 shutdown_connections 2024-03-29T11:34:38.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.403+0000 7f13e498c640 1 -- 172.21.15.71:0/2154287545 >> 172.21.15.71:0/2154287545 conn(0x7f13e00b35e0 msgr2=0x7f13e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.404+0000 7f13e498c640 1 -- 172.21.15.71:0/2154287545 shutdown_connections 2024-03-29T11:34:38.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.404+0000 7f13e498c640 1 -- 172.21.15.71:0/2154287545 wait complete. 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.405+0000 7f13e498c640 10 monclient: build_initial_monmap 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.405+0000 7f13e498c640 10 monclient: monmap: 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 1 librados: starting msgr at 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 1 librados: starting objecter 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 1 Processor -- start 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 1 -- start start 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 1 librados: setting wanted keys 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 1 librados: calling monclient init 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.406+0000 7f13e498c640 10 monclient: init 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient(hunting): picked mon.noname-c con 0x7f13e00a9930 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient(hunting): picked mon.noname-a con 0x7f13e0149dc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient(hunting): picked mon.noname-b con 0x7f13e014ff40 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b3040 con 0x7f13e0149dc0 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e01de980 con 0x7f13e014ff40 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e01dec20 con 0x7f13e00a9930 2024-03-29T11:34:38.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:38.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.407+0000 7f13e498c640 10 monclient(hunting): authenticate will time out at 932.561234s 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.411+0000 7f13de59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f13e00a9930 0x7f13e00b2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48467/0 (socket says 172.21.15.71:48467) 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.411+0000 7f13de59c640 1 -- 172.21.15.71:0/14689238 learned_addr learned my addr 172.21.15.71:0/14689238 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.411+0000 7f13ceffd640 10 client.?.objecter ms_handle_connect 0x7f13e00a9930 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.411+0000 7f13ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181771835 0 0) 0x7f13e01dec20 con 0x7f13e00a9930 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:34:38.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0004eb0 con 0x7f13e00a9930 2024-03-29T11:34:38.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 client.?.objecter ms_handle_connect 0x7f13e0149dc0 2024-03-29T11:34:38.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1919336044 0 0) 0x7f13c0004eb0 con 0x7f13e00a9930 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e01dec20 con 0x7f13e00a9930 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f13d0001960 con 0x7f13e00a9930 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): dump: 2024-03-29T11:34:38.412 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282492114 0 0) 0x7f13e00b3040 con 0x7f13e0149dc0 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0004eb0 con 0x7f13e0149dc0 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 client.?.objecter ms_handle_connect 0x7f13e014ff40 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.412+0000 7f13ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353988894 0 0) 0x7f13e01de980 con 0x7f13e014ff40 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e00b3040 con 0x7f13e014ff40 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1538731350 0 0) 0x7f13c0004eb0 con 0x7f13e0149dc0 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e01de980 con 0x7f13e0149dc0 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f13d4001510 con 0x7f13e0149dc0 2024-03-29T11:34:38.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): dump: 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318952342 0 0) 0x7f13e00b3040 con 0x7f13e014ff40 2024-03-29T11:34:38.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13c0004eb0 con 0x7f13e014ff40 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f13d8004d00 con 0x7f13e014ff40 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.413+0000 7f13ceffd640 10 monclient(hunting): dump: 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:38.416 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.414+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3507730131 0 0) 0x7f13c0004eb0 con 0x7f13e014ff40 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 >> v1:172.21.15.71:6790/0 conn(0x7f13e00a9930 legacy=0x7f13e00b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 >> v1:172.21.15.71:6789/0 conn(0x7f13e0149dc0 legacy=0x7f13e01df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 1 monclient: found mon.b 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e01df850 con 0x7f13e014ff40 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:38.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 10 monclient: _renew_subs 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e0149890 con 0x7f13e014ff40 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 10 monclient: _renew_subs 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e00b1d90 con 0x7f13e014ff40 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13e498c640 1 librados: init done 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d8001e50 con 0x7f13e014ff40 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.415+0000 7f13ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.416+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f13d80052b0 con 0x7f13e014ff40 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.416+0000 7f13ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.416+0000 7f13ceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:38.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.416+0000 7f13ceffd640 10 monclient: dump: 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.416+0000 7f13dcd99640 4 set_mon_vals no callback set 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.418+0000 7f13e498c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.418+0000 7f13e498c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:38.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.418+0000 7f13e498c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.418+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4005080 con 0x7f13e014ff40 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.419+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f13d801b280 con 0x7f13e014ff40 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.419+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f13d8084e40 con 0x7f13e014ff40 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.419+0000 7f13ceffd640 10 client.4330.objecter ms_dispatch 0x7f13e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.419+0000 7f13ceffd640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.419+0000 7f13ceffd640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.419+0000 7f13ceffd640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.420+0000 7f13de59c640 10 monclient: get_auth_request con 0x7f13c006e100 auth_method 0 2024-03-29T11:34:38.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.420+0000 7f13ceffd640 10 client.4330.objecter ms_handle_connect 0x7f13c006e100 2024-03-29T11:34:38.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.427+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13d8050720 con 0x7f13e014ff40 2024-03-29T11:34:38.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.427+0000 7f13ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:38.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.427+0000 7f13ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:38.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.747+0000 7f13e498c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:38.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.748+0000 7f13e498c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:38.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.748+0000 7f13e498c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:38.745 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.748+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13c4005330 con 0x7f13e014ff40 2024-03-29T11:34:38.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.751+0000 7f13ceffd640 1 -- 172.21.15.71:0/14689238 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f13d8052650 con 0x7f13e014ff40 2024-03-29T11:34:38.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.751+0000 7f13ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:38.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.751+0000 7f13ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:38.749 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:38.750 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:38.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.757+0000 7f13e498c640 10 librados: watch_flush enter 2024-03-29T11:34:38.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 10 librados: watch_flush exit 2024-03-29T11:34:38.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 >> v1:172.21.15.71:6816/1596514235 conn(0x7f13c006e100 legacy=0x7f13c00705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 10 monclient: shutdown 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 >> v1:172.21.15.81:6789/0 conn(0x7f13e014ff40 legacy=0x7f13e0149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 shutdown_connections 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 >> 172.21.15.71:0/14689238 conn(0x7f13e00b35e0 msgr2=0x7f13e00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 shutdown_connections 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.759+0000 7f13e498c640 1 -- 172.21.15.71:0/14689238 wait complete. 2024-03-29T11:34:38.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:38.760+0000 7f13e498c640 1 librados: shutdown 2024-03-29T11:34:38.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:45.582 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:45.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.739+0000 7fa40e5ad640 10 monclient: get_monmap_and_config 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.739+0000 7fa40e5ad640 10 monclient: build_initial_monmap 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.739+0000 7fa40e5ad640 10 monclient: monmap: 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:45.740385+0000 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:45.740385+0000 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:45.738 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:45.739 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.740+0000 7fa40e5ad640 1 Processor -- start 2024-03-29T11:34:45.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.741+0000 7fa40e5ad640 1 -- start start 2024-03-29T11:34:45.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.741+0000 7fa40e5ad640 10 monclient: init 2024-03-29T11:34:45.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.742+0000 7fa40e5ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:45.740 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.742+0000 7fa40e5ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 10 monclient(hunting): picked mon.noname-a con 0x7fa400034850 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 10 monclient(hunting): picked mon.noname-c con 0x7fa4000c4ec0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 10 monclient(hunting): picked mon.noname-b con 0x7fa4000c7d30 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4000c83c0 con 0x7fa400034850 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa400033280 con 0x7fa4000c4ec0 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa400029100 con 0x7fa4000c7d30 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:45.741 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.743+0000 7fa40e5ad640 10 monclient(hunting): authenticate will time out at 939.897677s 2024-03-29T11:34:45.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.746+0000 7fa40d5ab640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fa400034850 0x7fa4000c47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47931/0 (socket says 172.21.15.71:47931) 2024-03-29T11:34:45.744 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.746+0000 7fa40d5ab640 1 -- 172.21.15.71:0/2824294260 learned_addr learned my addr 172.21.15.71:0/2824294260 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.748+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26134157 0 0) 0x7fa400033280 con 0x7fa4000c4ec0 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.748+0000 7fa3fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.748+0000 7fa3fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.748+0000 7fa3fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.748+0000 7fa3fffff640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.748+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3f4003520 con 0x7fa4000c4ec0 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667885529 0 0) 0x7fa3f4003520 con 0x7fa4000c4ec0 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 >> v1:172.21.15.71:6790/0 conn(0x7fa4000c7d30 legacy=0x7fa400029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 >> v1:172.21.15.71:6789/0 conn(0x7fa400034850 legacy=0x7fa4000c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 1 monclient: found mon.noname-c 2024-03-29T11:34:45.748 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000293e0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 10 monclient: _finish_auth 0 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.749+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa3f8001690 con 0x7fa4000c4ec0 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: dump: 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.749 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa40e5ad640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa40e5ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3f80017f0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 1 -- 172.21.15.71:0/2824294260 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa3f8001c00 con 0x7fa4000c4ec0 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa3fffff640 10 monclient: dump: 2024-03-29T11:34:45.750 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa40e5ad640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:45.751 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.750+0000 7fa40e5ad640 4 set_mon_vals no callback set 2024-03-29T11:34:45.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.751+0000 7fa40e5ad640 10 monclient: shutdown 2024-03-29T11:34:45.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.751+0000 7fa40e5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:45.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.751+0000 7fa40e5ad640 1 -- 172.21.15.71:0/2824294260 >> v1:172.21.15.81:6789/0 conn(0x7fa4000c4ec0 legacy=0x7fa400029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:45.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.754+0000 7fa40e5ad640 1 -- 172.21.15.71:0/2824294260 shutdown_connections 2024-03-29T11:34:45.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.754+0000 7fa40e5ad640 1 -- 172.21.15.71:0/2824294260 >> 172.21.15.71:0/2824294260 conn(0x7fa4000058f0 msgr2=0x7fa400005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:45.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.755+0000 7fa40e5ad640 1 -- 172.21.15.71:0/2824294260 shutdown_connections 2024-03-29T11:34:45.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.755+0000 7fa40e5ad640 1 -- 172.21.15.71:0/2824294260 wait complete. 2024-03-29T11:34:45.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.756+0000 7fa40e5ad640 10 monclient: build_initial_monmap 2024-03-29T11:34:45.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.756+0000 7fa40e5ad640 10 monclient: monmap: 2024-03-29T11:34:45.754 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:45.754 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.756+0000 7fa40e5ad640 1 librados: starting msgr at 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.756+0000 7fa40e5ad640 1 librados: starting objecter 2024-03-29T11:34:45.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.756+0000 7fa40e5ad640 1 Processor -- start 2024-03-29T11:34:45.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.757+0000 7fa40e5ad640 1 -- start start 2024-03-29T11:34:45.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.757+0000 7fa40e5ad640 1 librados: setting wanted keys 2024-03-29T11:34:45.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.757+0000 7fa40e5ad640 1 librados: calling monclient init 2024-03-29T11:34:45.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.757+0000 7fa40e5ad640 10 monclient: init 2024-03-29T11:34:45.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.757+0000 7fa40e5ad640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:45.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.757+0000 7fa40e5ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 10 monclient(hunting): picked mon.noname-c con 0x7fa400034850 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 10 monclient(hunting): picked mon.noname-b con 0x7fa4000c4ec0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 10 monclient(hunting): picked mon.noname-a con 0x7fa4000c7d30 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4000ca2a0 con 0x7fa4000c7d30 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4000caff0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4000cb260 con 0x7fa400034850 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.760+0000 7fa40e5ad640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.761+0000 7fa40e5ad640 10 monclient(hunting): authenticate will time out at 939.915058s 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.764+0000 7fa40cdaa640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa4000c4ec0 0x7fa4000bf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:56651/0 (socket says 172.21.15.71:56651) 2024-03-29T11:34:45.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.764+0000 7fa40cdaa640 1 -- 172.21.15.71:0/1599509662 learned_addr learned my addr 172.21.15.71:0/1599509662 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 10 client.?.objecter ms_handle_connect 0x7fa4000c4ec0 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055057846 0 0) 0x7fa4000caff0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.767+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3ec004eb0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 client.?.objecter ms_handle_connect 0x7fa4000c7d30 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 client.?.objecter ms_handle_connect 0x7fa400034850 2024-03-29T11:34:45.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756018933 0 0) 0x7fa4000cb260 con 0x7fa400034850 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.768+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4000caff0 con 0x7fa400034850 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3116822725 0 0) 0x7fa3ec004eb0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4000cb260 con 0x7fa4000c4ec0 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa3f8001b00 con 0x7fa4000c4ec0 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:45.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): dump: 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481209885 0 0) 0x7fa4000caff0 con 0x7fa400034850 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3ec004eb0 con 0x7fa400034850 2024-03-29T11:34:45.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215848771 0 0) 0x7fa4000ca2a0 con 0x7fa4000c7d30 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4000caff0 con 0x7fa4000c7d30 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa40804fdd0 con 0x7fa400034850 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.769+0000 7fa3fdffb640 10 monclient(hunting): dump: 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.774 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 298512991 0 0) 0x7fa4000cb260 con 0x7fa4000c4ec0 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 >> v1:172.21.15.71:6790/0 conn(0x7fa400034850 legacy=0x7fa4000b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 >> v1:172.21.15.71:6789/0 conn(0x7fa4000c7d30 legacy=0x7fa4000c0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 monclient: found mon.b 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:45.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000cb540 con 0x7fa4000c4ec0 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa40e5ad640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3f80038a0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa3f8003cb0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3fdffb640 10 monclient: dump: 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:45.776 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.770+0000 7fa3feffd640 4 set_mon_vals no callback set 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.771+0000 7fa40e5ad640 10 monclient: _renew_subs 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.771+0000 7fa40e5ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.771+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4000c8ee0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.772+0000 7fa40e5ad640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.772+0000 7fa40e5ad640 10 monclient: _renew_subs 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.772+0000 7fa40e5ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.772+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4000cbb40 con 0x7fa4000c4ec0 2024-03-29T11:34:45.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.772+0000 7fa40e5ad640 1 librados: init done 2024-03-29T11:34:45.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.774+0000 7fa40e5ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.774+0000 7fa40e5ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.774+0000 7fa40e5ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.774+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa400004cd0 con 0x7fa4000c4ec0 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.776+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fa3f8003e10 con 0x7fa4000c4ec0 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.779+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fa3f8084b20 con 0x7fa4000c4ec0 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.779+0000 7fa3fdffb640 10 client.4336.objecter ms_dispatch 0x7fa408000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.779+0000 7fa3fdffb640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.779+0000 7fa3fdffb640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.779+0000 7fa3fdffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.779+0000 7fa40d5ab640 10 monclient: get_auth_request con 0x7fa3ec06de20 auth_method 0 2024-03-29T11:34:45.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.780+0000 7fa3fdffb640 10 client.4336.objecter ms_handle_connect 0x7fa3ec06de20 2024-03-29T11:34:45.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.784+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3f8050400 con 0x7fa4000c4ec0 2024-03-29T11:34:45.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.784+0000 7fa3fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:45.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:45.784+0000 7fa3fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:46.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.115+0000 7fa40e5ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:46.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.115+0000 7fa40e5ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:46.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.115+0000 7fa40e5ad640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:34:46.114 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.115+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa400038a30 con 0x7fa4000c4ec0 2024-03-29T11:34:46.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.116+0000 7fa3fdffb640 1 -- 172.21.15.71:0/1599509662 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fa3f8052330 con 0x7fa4000c4ec0 2024-03-29T11:34:46.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.116+0000 7fa3fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:46.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.116+0000 7fa3fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:46.115 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:46.115 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:46.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 10 librados: watch_flush enter 2024-03-29T11:34:46.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 10 librados: watch_flush exit 2024-03-29T11:34:46.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:34:46.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:34:46.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 >> v1:172.21.15.71:6816/1596514235 conn(0x7fa3ec06de20 legacy=0x7fa3ec0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:46.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 10 monclient: shutdown 2024-03-29T11:34:46.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:46.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 >> v1:172.21.15.81:6789/0 conn(0x7fa4000c4ec0 legacy=0x7fa4000bf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:46.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 shutdown_connections 2024-03-29T11:34:46.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.120+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 >> 172.21.15.71:0/1599509662 conn(0x7fa4000058f0 msgr2=0x7fa4000c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:46.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.121+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 shutdown_connections 2024-03-29T11:34:46.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.122+0000 7fa40e5ad640 1 -- 172.21.15.71:0/1599509662 wait complete. 2024-03-29T11:34:46.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:46.122+0000 7fa40e5ad640 1 librados: shutdown 2024-03-29T11:34:46.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:52.944 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:53.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.115+0000 7f8db0b13640 10 monclient: get_monmap_and_config 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.115+0000 7f8db0b13640 10 monclient: build_initial_monmap 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.115+0000 7f8db0b13640 10 monclient: monmap: 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:34:53.115966+0000 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:34:53.115966+0000 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:34:53.113 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:53.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.118+0000 7f8db0b13640 1 Processor -- start 2024-03-29T11:34:53.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.118+0000 7f8db0b13640 1 -- start start 2024-03-29T11:34:53.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.118+0000 7f8db0b13640 10 monclient: init 2024-03-29T11:34:53.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.119+0000 7f8db0b13640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:53.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.119+0000 7f8db0b13640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:53.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.119+0000 7f8db0b13640 10 monclient(hunting): picked mon.noname-c con 0x7f8dac1422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 10 monclient(hunting): picked mon.noname-b con 0x7f8dac142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 10 monclient(hunting): picked mon.noname-a con 0x7f8dac0b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac142c80 con 0x7f8dac0b17d0 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac0aae40 con 0x7f8dac1422b0 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac13fbd0 con 0x7f8dac142690 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8db0b13640 10 monclient(hunting): authenticate will time out at 947.277635s 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8da9d9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8dac142690 0x7f8dac0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46119/0 (socket says 172.21.15.71:46119) 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.123+0000 7f8da9d9b640 1 -- 172.21.15.71:0/846680751 learned_addr learned my addr 172.21.15.71:0/846680751 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080111805 0 0) 0x7f8dac13fbd0 con 0x7f8dac142690 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:53.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:53.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:34:53.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d94003520 con 0x7f8dac142690 2024-03-29T11:34:53.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104792979 0 0) 0x7f8d94003520 con 0x7f8dac142690 2024-03-29T11:34:53.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 >> v1:172.21.15.81:6789/0 conn(0x7f8dac1422b0 legacy=0x7f8dac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:53.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.124+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 >> v1:172.21.15.71:6789/0 conn(0x7f8dac0b17d0 legacy=0x7f8dac0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 1 monclient: found mon.noname-b 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac151940 con 0x7f8dac142690 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: _finish_auth 0 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8da0001690 con 0x7f8dac142690 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:53.127 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.127+0000 7f8da959a640 10 monclient: dump: 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8db0b13640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8db0b13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8da00018b0 con 0x7f8dac142690 2024-03-29T11:34:53.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8da959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8da959a640 1 -- 172.21.15.71:0/846680751 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8da0001c40 con 0x7f8dac142690 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8da959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8da959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8da959a640 10 monclient: dump: 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:53.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8db0b13640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:53.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.128+0000 7f8db0b13640 4 set_mon_vals no callback set 2024-03-29T11:34:53.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.129+0000 7f8db0b13640 10 monclient: shutdown 2024-03-29T11:34:53.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.129+0000 7f8db0b13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:53.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.129+0000 7f8db0b13640 1 -- 172.21.15.71:0/846680751 >> v1:172.21.15.71:6790/0 conn(0x7f8dac142690 legacy=0x7f8dac0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:53.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.129+0000 7f8db0b13640 1 -- 172.21.15.71:0/846680751 shutdown_connections 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.129+0000 7f8db0b13640 1 -- 172.21.15.71:0/846680751 >> 172.21.15.71:0/846680751 conn(0x7f8dac066a00 msgr2=0x7f8dac066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.131+0000 7f8db0b13640 1 -- 172.21.15.71:0/846680751 shutdown_connections 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.131+0000 7f8db0b13640 1 -- 172.21.15.71:0/846680751 wait complete. 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.132+0000 7f8db0b13640 10 monclient: build_initial_monmap 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.132+0000 7f8db0b13640 10 monclient: monmap: 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:34:53.131 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.133+0000 7f8db0b13640 1 librados: starting msgr at 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.133+0000 7f8db0b13640 1 librados: starting objecter 2024-03-29T11:34:53.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.133+0000 7f8db0b13640 1 Processor -- start 2024-03-29T11:34:53.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.133+0000 7f8db0b13640 1 -- start start 2024-03-29T11:34:53.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.133+0000 7f8db0b13640 1 librados: setting wanted keys 2024-03-29T11:34:53.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.133+0000 7f8db0b13640 1 librados: calling monclient init 2024-03-29T11:34:53.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.134+0000 7f8db0b13640 10 monclient: init 2024-03-29T11:34:53.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.134+0000 7f8db0b13640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:53.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.134+0000 7f8db0b13640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:53.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.135+0000 7f8db0b13640 10 monclient(hunting): picked mon.noname-c con 0x7f8dac0b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:34:53.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 10 monclient(hunting): picked mon.noname-a con 0x7f8dac1422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 10 monclient(hunting): picked mon.noname-b con 0x7f8dac142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac0b12e0 con 0x7f8dac1422b0 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac0b1000 con 0x7f8dac142690 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac12e900 con 0x7f8dac0b17d0 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.137+0000 7f8db0b13640 10 monclient(hunting): authenticate will time out at 947.291854s 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8daa59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8dac0b17d0 0x7f8dac131c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46199/0 (socket says 172.21.15.71:46199) 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8daa59c640 1 -- 172.21.15.71:0/1917485272 learned_addr learned my addr 172.21.15.71:0/1917485272 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 10 client.?.objecter ms_handle_connect 0x7f8dac0b17d0 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:53.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519200424 0 0) 0x7f8dac12e900 con 0x7f8dac0b17d0 2024-03-29T11:34:53.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.138+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8c004eb0 con 0x7f8dac0b17d0 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1814335287 0 0) 0x7f8d8c004eb0 con 0x7f8dac0b17d0 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dac12e900 con 0x7f8dac0b17d0 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8d9c001990 con 0x7f8dac0b17d0 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:53.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 10 monclient(hunting): dump: 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.141+0000 7f8d9affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:53.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 client.?.objecter ms_handle_connect 0x7f8dac142690 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 client.?.objecter ms_handle_connect 0x7f8dac1422b0 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626669054 0 0) 0x7f8dac0b1000 con 0x7f8dac142690 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:34:53.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8c004eb0 con 0x7f8dac142690 2024-03-29T11:34:53.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2950578527 0 0) 0x7f8dac12e900 con 0x7f8dac0b17d0 2024-03-29T11:34:53.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 >> v1:172.21.15.81:6789/0 conn(0x7f8dac142690 legacy=0x7f8dac12e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:53.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 >> v1:172.21.15.71:6789/0 conn(0x7f8dac1422b0 legacy=0x7f8dac12dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:53.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:53.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 monclient: found mon.c 2024-03-29T11:34:53.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac0ac160 con 0x7f8dac0b17d0 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient: _finish_auth 0 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.142+0000 7f8d9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.143+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d9c001c90 con 0x7f8dac0b17d0 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.143+0000 7f8d9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.143+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8d9c003a60 con 0x7f8dac0b17d0 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.143+0000 7f8d9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.143+0000 7f8d9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.143+0000 7f8d9affd640 10 monclient: dump: 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:34:53.146 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.145+0000 7f8da8d99640 4 set_mon_vals no callback set 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.145+0000 7f8db0b13640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.145+0000 7f8db0b13640 10 monclient: _renew_subs 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.145+0000 7f8db0b13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.145+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dac0ac440 con 0x7f8dac0b17d0 2024-03-29T11:34:53.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.146+0000 7f8db0b13640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:53.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.146+0000 7f8db0b13640 10 monclient: _renew_subs 2024-03-29T11:34:53.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.146+0000 7f8db0b13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:53.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.146+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dac132b40 con 0x7f8dac0b17d0 2024-03-29T11:34:53.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.146+0000 7f8db0b13640 1 librados: init done 2024-03-29T11:34:53.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.149+0000 7f8db0b13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:53.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.149+0000 7f8db0b13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:53.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.149+0000 7f8db0b13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:53.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.150+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dac075f10 con 0x7f8dac0b17d0 2024-03-29T11:34:53.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.150+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8d9c003bc0 con 0x7f8dac0b17d0 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8da9d9b640 10 monclient: get_auth_request con 0x7f8d8c0764c0 auth_method 0 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f8d9c084a60 con 0x7f8dac0b17d0 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8d9affd640 10 client.4352.objecter ms_dispatch 0x7f8dac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8d9affd640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8d9affd640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8d9affd640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:34:53.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.151+0000 7f8d9affd640 10 client.4352.objecter ms_handle_connect 0x7f8d8c0764c0 2024-03-29T11:34:53.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.158+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d9c050340 con 0x7f8dac0b17d0 2024-03-29T11:34:53.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.158+0000 7f8d9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:53.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.158+0000 7f8d9affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:53.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.480+0000 7f8db0b13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:53.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.480+0000 7f8db0b13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:53.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.480+0000 7f8db0b13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:34:53.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.480+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dac0b2270 con 0x7f8dac0b17d0 2024-03-29T11:34:53.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.483+0000 7f8d9affd640 1 -- 172.21.15.71:0/1917485272 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f8d9c052270 con 0x7f8dac0b17d0 2024-03-29T11:34:53.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.483+0000 7f8d9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:53.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.483+0000 7f8d9affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:53.481 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:34:53.481 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:53.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 10 librados: watch_flush enter 2024-03-29T11:34:53.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 10 librados: watch_flush exit 2024-03-29T11:34:53.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:34:53.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:34:53.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 >> v1:172.21.15.71:6816/1596514235 conn(0x7f8d8c0764c0 legacy=0x7f8d8c078980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:53.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 10 monclient: shutdown 2024-03-29T11:34:53.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:53.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.487+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 >> v1:172.21.15.71:6790/0 conn(0x7f8dac0b17d0 legacy=0x7f8dac131c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:53.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.488+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 shutdown_connections 2024-03-29T11:34:53.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.488+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 >> 172.21.15.71:0/1917485272 conn(0x7f8dac066a00 msgr2=0x7f8dac0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:53.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.488+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 shutdown_connections 2024-03-29T11:34:53.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.488+0000 7f8db0b13640 1 -- 172.21.15.71:0/1917485272 wait complete. 2024-03-29T11:34:53.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:34:53.489+0000 7f8db0b13640 1 librados: shutdown 2024-03-29T11:34:53.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:00.309 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.463+0000 7fea8bdd0640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.463+0000 7fea8bdd0640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.463+0000 7fea8bdd0640 10 monclient: monmap: 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:00.464458+0000 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:00.464458+0000 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:00.463 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:00.464 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:00.464 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:00.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.466+0000 7fea8bdd0640 1 Processor -- start 2024-03-29T11:35:00.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.466+0000 7fea8bdd0640 1 -- start start 2024-03-29T11:35:00.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.466+0000 7fea8bdd0640 10 monclient: init 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient(hunting): picked mon.noname-a con 0x7fea840a9ac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient(hunting): picked mon.noname-b con 0x7fea8413ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient(hunting): picked mon.noname-c con 0x7fea840af900 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840afd10 con 0x7fea840a9ac0 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840af6f0 con 0x7fea840af900 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea840aac80 con 0x7fea8413ee90 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.467+0000 7fea8bdd0640 10 monclient(hunting): authenticate will time out at 954.621979s 2024-03-29T11:35:00.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.473+0000 7fea8a36d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fea840af900 0x7fea84149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:50109/0 (socket says 172.21.15.71:50109) 2024-03-29T11:35:00.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.473+0000 7fea8a36d640 1 -- 172.21.15.71:0/3921605083 learned_addr learned my addr 172.21.15.71:0/3921605083 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:00.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.473+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226251150 0 0) 0x7fea840aac80 con 0x7fea8413ee90 2024-03-29T11:35:00.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.471 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea70003520 con 0x7fea8413ee90 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605160102 0 0) 0x7fea840af6f0 con 0x7fea840af900 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea840aac80 con 0x7fea840af900 2024-03-29T11:35:00.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 239928653 0 0) 0x7fea840aac80 con 0x7fea840af900 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 >> v1:172.21.15.71:6790/0 conn(0x7fea8413ee90 legacy=0x7fea8413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 >> v1:172.21.15.71:6789/0 conn(0x7fea840a9ac0 legacy=0x7fea840adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea840aaf60 con 0x7fea840af900 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fea7c0019f0 con 0x7fea840af900 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: dump: 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea7c001c40 con 0x7fea840af900 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 1 -- 172.21.15.71:0/3921605083 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fea7c001fd0 con 0x7fea840af900 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.474+0000 7fea88b6a640 10 monclient: dump: 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:00.475 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:00.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.477+0000 7fea8bdd0640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:35:00.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.477+0000 7fea8bdd0640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.477+0000 7fea8bdd0640 4 set_mon_vals no callback set 2024-03-29T11:35:00.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.478+0000 7fea8bdd0640 10 monclient: shutdown 2024-03-29T11:35:00.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.478+0000 7fea8bdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.478+0000 7fea8bdd0640 1 -- 172.21.15.71:0/3921605083 >> v1:172.21.15.81:6789/0 conn(0x7fea840af900 legacy=0x7fea84149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.480+0000 7fea8bdd0640 1 -- 172.21.15.71:0/3921605083 shutdown_connections 2024-03-29T11:35:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.480+0000 7fea8bdd0640 1 -- 172.21.15.71:0/3921605083 >> 172.21.15.71:0/3921605083 conn(0x7fea840b35e0 msgr2=0x7fea840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.482+0000 7fea8bdd0640 1 -- 172.21.15.71:0/3921605083 shutdown_connections 2024-03-29T11:35:00.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.482+0000 7fea8bdd0640 1 -- 172.21.15.71:0/3921605083 wait complete. 2024-03-29T11:35:00.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.483+0000 7fea8bdd0640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.483+0000 7fea8bdd0640 10 monclient: monmap: 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:00.481 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:00.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.483+0000 7fea8bdd0640 1 librados: starting msgr at 2024-03-29T11:35:00.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.483+0000 7fea8bdd0640 1 librados: starting objecter 2024-03-29T11:35:00.482 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.483+0000 7fea8bdd0640 1 Processor -- start 2024-03-29T11:35:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.484+0000 7fea8bdd0640 1 -- start start 2024-03-29T11:35:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.484+0000 7fea8bdd0640 1 librados: setting wanted keys 2024-03-29T11:35:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.484+0000 7fea8bdd0640 1 librados: calling monclient init 2024-03-29T11:35:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.484+0000 7fea8bdd0640 10 monclient: init 2024-03-29T11:35:00.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:00.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient(hunting): picked mon.noname-a con 0x7fea840a9ac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:00.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient(hunting): picked mon.noname-b con 0x7fea840af900 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient(hunting): picked mon.noname-c con 0x7fea8413ee90 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea841deab0 con 0x7fea840a9ac0 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea841df650 con 0x7fea840af900 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea841410d0 con 0x7fea8413ee90 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.485+0000 7fea8bdd0640 10 monclient(hunting): authenticate will time out at 954.640040s 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.490+0000 7fea89b6c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fea840a9ac0 0x7fea841def40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35935/0 (socket says 172.21.15.71:35935) 2024-03-29T11:35:00.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.490+0000 7fea89b6c640 1 -- 172.21.15.71:0/466116575 learned_addr learned my addr 172.21.15.71:0/466116575 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:00.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.491+0000 7fea7a7fc640 10 client.?.objecter ms_handle_connect 0x7fea840a9ac0 2024-03-29T11:35:00.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.491+0000 7fea7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 10 client.?.objecter ms_handle_connect 0x7fea8413ee90 2024-03-29T11:35:00.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19556282 0 0) 0x7fea841410d0 con 0x7fea8413ee90 2024-03-29T11:35:00.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.492+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea68004eb0 con 0x7fea8413ee90 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239514996 0 0) 0x7fea68004eb0 con 0x7fea8413ee90 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea841410d0 con 0x7fea8413ee90 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fea7c0019b0 con 0x7fea8413ee90 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.492 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2486442316 0 0) 0x7fea841410d0 con 0x7fea8413ee90 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 >> v1:172.21.15.81:6789/0 conn(0x7fea840af900 legacy=0x7fea841409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 >> v1:172.21.15.71:6789/0 conn(0x7fea840a9ac0 legacy=0x7fea841def40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 1 monclient: found mon.c 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.493+0000 7fea7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:00.493 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea84141260 con 0x7fea8413ee90 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea7c004e10 con 0x7fea8413ee90 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fea7c005120 con 0x7fea8413ee90 2024-03-29T11:35:00.494 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7a7fc640 10 monclient: dump: 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:35:00.495 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 10 monclient: _renew_subs 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8413c0c0 con 0x7fea8413ee90 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 10 monclient: _renew_subs 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8413c410 con 0x7fea8413ee90 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea8bdd0640 1 librados: init done 2024-03-29T11:35:00.496 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.494+0000 7fea7bfff640 4 set_mon_vals no callback set 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.498+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fea7c01b0f0 con 0x7fea8413ee90 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.498+0000 7fea8bdd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.498+0000 7fea8bdd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.498+0000 7fea8bdd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.498+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fea7c084ec0 con 0x7fea8413ee90 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.498+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea84135f00 con 0x7fea8413ee90 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.499+0000 7fea7a7fc640 10 client.4358.objecter ms_dispatch 0x7fea84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.499+0000 7fea7a7fc640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.499+0000 7fea7a7fc640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:00.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.499+0000 7fea7a7fc640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.499+0000 7fea89b6c640 10 monclient: get_auth_request con 0x7fea6806db90 auth_method 0 2024-03-29T11:35:00.498 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.499+0000 7fea7a7fc640 10 client.4358.objecter ms_handle_connect 0x7fea6806db90 2024-03-29T11:35:00.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.508+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea7c050800 con 0x7fea8413ee90 2024-03-29T11:35:00.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.508+0000 7fea7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.506 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.508+0000 7fea7a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.832+0000 7fea8bdd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:00.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.832+0000 7fea8bdd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:00.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.832+0000 7fea8bdd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:00.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.832+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea84075f10 con 0x7fea8413ee90 2024-03-29T11:35:00.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.833+0000 7fea7a7fc640 1 -- 172.21.15.71:0/466116575 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fea7c050800 con 0x7fea8413ee90 2024-03-29T11:35:00.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.833+0000 7fea7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:00.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.833+0000 7fea7a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:00.832 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:00.832 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:00.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.838+0000 7fea8bdd0640 10 librados: watch_flush enter 2024-03-29T11:35:00.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.838+0000 7fea8bdd0640 10 librados: watch_flush exit 2024-03-29T11:35:00.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.838+0000 7fea8bdd0640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.838+0000 7fea8bdd0640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:35:00.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.839+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 >> v1:172.21.15.71:6816/1596514235 conn(0x7fea6806db90 legacy=0x7fea68070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.839+0000 7fea8bdd0640 10 monclient: shutdown 2024-03-29T11:35:00.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.839+0000 7fea8bdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.839+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 >> v1:172.21.15.71:6790/0 conn(0x7fea8413ee90 legacy=0x7fea841354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.840+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 shutdown_connections 2024-03-29T11:35:00.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.840+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 >> 172.21.15.71:0/466116575 conn(0x7fea840b35e0 msgr2=0x7fea841ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.841+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 shutdown_connections 2024-03-29T11:35:00.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.841+0000 7fea8bdd0640 1 -- 172.21.15.71:0/466116575 wait complete. 2024-03-29T11:35:00.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:00.841+0000 7fea8bdd0640 1 librados: shutdown 2024-03-29T11:35:00.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:07.667 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:07.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.813+0000 7f8621438640 10 monclient: get_monmap_and_config 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.813+0000 7f8621438640 10 monclient: build_initial_monmap 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.813+0000 7f8621438640 10 monclient: monmap: 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:07.814721+0000 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:07.814721+0000 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:07.812 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:07.813 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.814+0000 7f8621438640 1 Processor -- start 2024-03-29T11:35:07.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.815+0000 7f8621438640 1 -- start start 2024-03-29T11:35:07.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.815+0000 7f8621438640 10 monclient: init 2024-03-29T11:35:07.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.816+0000 7f8621438640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:07.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.816+0000 7f8621438640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:07.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.819+0000 7f8621438640 10 monclient(hunting): picked mon.noname-b con 0x7f861c0a9750 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:07.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.819+0000 7f861bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f861c0a9750 0x7f861c0ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39261/0 (socket says 172.21.15.71:39261) 2024-03-29T11:35:07.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.819+0000 7f861bfff640 1 -- 172.21.15.71:0/2936975723 learned_addr learned my addr 172.21.15.71:0/2936975723 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:07.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.819+0000 7f8621438640 10 monclient(hunting): picked mon.noname-a con 0x7f861c0aad20 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.819+0000 7f8621438640 10 monclient(hunting): picked mon.noname-c con 0x7f861c0a9b30 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0af4d0 con 0x7f861c0aad20 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0af7b0 con 0x7f861c0a9b30 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c0acab0 con 0x7f861c0a9750 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f8621438640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f8621438640 10 monclient(hunting): authenticate will time out at 961.974799s 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491320339 0 0) 0x7f861c0acab0 con 0x7f861c0a9750 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f861affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f861affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.819 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f861affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f861affd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:35:07.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.820+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8604003520 con 0x7f861c0a9750 2024-03-29T11:35:07.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.821+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275012712 0 0) 0x7f8604003520 con 0x7f861c0a9750 2024-03-29T11:35:07.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.821+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 >> v1:172.21.15.81:6789/0 conn(0x7f861c0a9b30 legacy=0x7f861c149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:07.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 >> v1:172.21.15.71:6789/0 conn(0x7f861c0aad20 legacy=0x7f861c141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 1 monclient: found mon.noname-b 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c0aba30 con 0x7f861c0a9750 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f860c001690 con 0x7f861c0a9750 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.823+0000 7f861affd640 10 monclient: dump: 2024-03-29T11:35:07.823 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.824+0000 7f8621438640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.824+0000 7f8621438640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f860c0017f0 con 0x7f861c0a9750 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f861affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:07.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f861affd640 1 -- 172.21.15.71:0/2936975723 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f860c001c40 con 0x7f861c0a9750 2024-03-29T11:35:07.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f861affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f861affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:07.825 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f861affd640 10 monclient: dump: 2024-03-29T11:35:07.825 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f8621438640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f8621438640 4 set_mon_vals no callback set 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f8621438640 10 monclient: shutdown 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f8621438640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:07.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.825+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 >> v1:172.21.15.71:6790/0 conn(0x7f861c0a9750 legacy=0x7f861c0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.826+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 shutdown_connections 2024-03-29T11:35:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.826+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 >> 172.21.15.71:0/2936975723 conn(0x7f861c0673e0 msgr2=0x7f861c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.826+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 shutdown_connections 2024-03-29T11:35:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.826+0000 7f8621438640 1 -- 172.21.15.71:0/2936975723 wait complete. 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.827+0000 7f8621438640 10 monclient: build_initial_monmap 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.827+0000 7f8621438640 10 monclient: monmap: 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.828+0000 7f8621438640 1 librados: starting msgr at 2024-03-29T11:35:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.828+0000 7f8621438640 1 librados: starting objecter 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.828+0000 7f8621438640 1 Processor -- start 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.829+0000 7f8621438640 1 -- start start 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.829+0000 7f8621438640 1 librados: setting wanted keys 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.829+0000 7f8621438640 1 librados: calling monclient init 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.829+0000 7f8621438640 10 monclient: init 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.829+0000 7f8621438640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.830+0000 7f8621438640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.830+0000 7f8621438640 10 monclient(hunting): picked mon.noname-a con 0x7f861c0a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:07.829 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.830+0000 7f8621438640 10 monclient(hunting): picked mon.noname-b con 0x7f861c0a9b30 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.830+0000 7f8621438640 10 monclient(hunting): picked mon.noname-c con 0x7f861c0aad20 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.830+0000 7f861bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f861c0a9750 0x7f861c1d31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:60542/0 (socket says 172.21.15.71:60542) 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f861bfff640 1 -- 172.21.15.71:0/921010078 learned_addr learned my addr 172.21.15.71:0/921010078 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c071a90 con 0x7f861c0a9750 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c1d2cb0 con 0x7f861c0a9b30 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f861c1d2910 con 0x7f861c0aad20 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8621438640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:07.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8621438640 10 monclient(hunting): authenticate will time out at 961.988683s 2024-03-29T11:35:07.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8618ff9640 10 client.?.objecter ms_handle_connect 0x7f861c0a9750 2024-03-29T11:35:07.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.834+0000 7f8618ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 client.?.objecter ms_handle_connect 0x7f861c0aad20 2024-03-29T11:35:07.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 client.?.objecter ms_handle_connect 0x7f861c0a9b30 2024-03-29T11:35:07.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568085519 0 0) 0x7f861c1d2910 con 0x7f861c0aad20 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.835+0000 7f8618ff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8608004eb0 con 0x7f861c0aad20 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918316510 0 0) 0x7f861c1d2cb0 con 0x7f861c0a9b30 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f861c1d2910 con 0x7f861c0a9b30 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1851444523 0 0) 0x7f861c1d2910 con 0x7f861c0a9b30 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f861c1d2cb0 con 0x7f861c0a9b30 2024-03-29T11:35:07.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2641955204 0 0) 0x7f8608004eb0 con 0x7f861c0aad20 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.836+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f861c1d2910 con 0x7f861c0aad20 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8610001700 con 0x7f861c0a9b30 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:07.838 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:07.839 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:07.839 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:07.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8614001af0 con 0x7f861c0aad20 2024-03-29T11:35:07.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.841 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055569893 0 0) 0x7f861c071a90 con 0x7f861c0a9750 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8608004eb0 con 0x7f861c0a9750 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3186311624 0 0) 0x7f861c1d2910 con 0x7f861c0aad20 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 >> v1:172.21.15.81:6789/0 conn(0x7f861c0a9b30 legacy=0x7f861c13ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 >> v1:172.21.15.71:6789/0 conn(0x7f861c0a9750 legacy=0x7f861c1d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 1 monclient: found mon.c 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.837+0000 7f8618ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8618ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c13b7c0 con 0x7f861c0aad20 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8618ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8618ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:07.843 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8618ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 10 monclient: _renew_subs 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c13ba70 con 0x7f861c0aad20 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 10 monclient: _renew_subs 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c13b430 con 0x7f861c0aad20 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.838+0000 7f8621438640 1 librados: init done 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.839+0000 7f8621438640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.839+0000 7f8621438640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.839+0000 7f8621438640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.839+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c1d2aa0 con 0x7f861c0aad20 2024-03-29T11:35:07.844 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.842+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8614001c50 con 0x7f861c0aad20 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.843+0000 7f8618ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.843+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8614003a10 con 0x7f861c0aad20 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.843+0000 7f8618ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.843+0000 7f8618ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.843+0000 7f8618ff9640 10 monclient: dump: 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:07.845 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.843+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8614019a60 con 0x7f861c0aad20 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f861a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f86140839c0 con 0x7f861c0aad20 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f8618ff9640 10 client.4364.objecter ms_dispatch 0x7f861c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f8618ff9640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f8618ff9640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f8618ff9640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:35:07.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.845+0000 7f861bfff640 10 monclient: get_auth_request con 0x7f860806dfa0 auth_method 0 2024-03-29T11:35:07.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.846+0000 7f8618ff9640 10 client.4364.objecter ms_handle_connect 0x7f860806dfa0 2024-03-29T11:35:07.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.849+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f861404f0d0 con 0x7f861c0aad20 2024-03-29T11:35:07.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.849+0000 7f8618ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:07.849+0000 7f8618ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:08.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.180+0000 7f8621438640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:08.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.180+0000 7f8621438640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:08.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.180+0000 7f8621438640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:08.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.180+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f861c04f730 con 0x7f861c0aad20 2024-03-29T11:35:08.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.182+0000 7f8618ff9640 1 -- 172.21.15.71:0/921010078 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f86140511d0 con 0x7f861c0aad20 2024-03-29T11:35:08.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.182+0000 7f8618ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:08.179 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.182+0000 7f8618ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:08.181 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:08.181 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:08.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.186+0000 7f8621438640 10 librados: watch_flush enter 2024-03-29T11:35:08.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.186+0000 7f8621438640 10 librados: watch_flush exit 2024-03-29T11:35:08.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.186+0000 7f8621438640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:35:08.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.186+0000 7f8621438640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:35:08.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.186+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 >> v1:172.21.15.71:6816/1596514235 conn(0x7f860806dfa0 legacy=0x7f8608070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 10 monclient: shutdown 2024-03-29T11:35:08.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:08.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 >> v1:172.21.15.71:6790/0 conn(0x7f861c0aad20 legacy=0x7f861c1de150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.185 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 shutdown_connections 2024-03-29T11:35:08.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 >> 172.21.15.71:0/921010078 conn(0x7f861c0673e0 msgr2=0x7f861c0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:08.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 shutdown_connections 2024-03-29T11:35:08.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 1 -- 172.21.15.71:0/921010078 wait complete. 2024-03-29T11:35:08.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:08.187+0000 7f8621438640 1 librados: shutdown 2024-03-29T11:35:08.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:15.007 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.155+0000 7fd536225640 10 monclient: get_monmap_and_config 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.155+0000 7fd536225640 10 monclient: build_initial_monmap 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.156+0000 7fd536225640 10 monclient: monmap: 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:15.156931+0000 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:15.156931+0000 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:15.154 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:15.155 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:15.155 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:15.155 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:15.155 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.157+0000 7fd536225640 1 Processor -- start 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.157+0000 7fd536225640 1 -- start start 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.157+0000 7fd536225640 10 monclient: init 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient(hunting): picked mon.noname-b con 0x7fd5301422b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:15.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient(hunting): picked mon.noname-c con 0x7fd530142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient(hunting): picked mon.noname-a con 0x7fd5300b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd530142c80 con 0x7fd5300b17d0 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300aae40 con 0x7fd530142690 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53013fbd0 con 0x7fd5301422b0 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd536225640 10 monclient(hunting): authenticate will time out at 969.312262s 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd52ffff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fd5300b17d0 0x7fd5300b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:57972/0 (socket says 172.21.15.71:57972) 2024-03-29T11:35:15.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.158+0000 7fd52ffff640 1 -- 172.21.15.71:0/3548287805 learned_addr learned my addr 172.21.15.71:0/3548287805 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192685735 0 0) 0x7fd5300aae40 con 0x7fd530142690 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd51c003520 con 0x7fd530142690 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123125991 0 0) 0x7fd530142c80 con 0x7fd5300b17d0 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.159+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5300aae40 con 0x7fd5300b17d0 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718207921 0 0) 0x7fd51c003520 con 0x7fd530142690 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 >> v1:172.21.15.71:6790/0 conn(0x7fd5301422b0 legacy=0x7fd53013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 >> v1:172.21.15.71:6789/0 conn(0x7fd5300b17d0 legacy=0x7fd5300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd530151940 con 0x7fd530142690 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:15.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd536225640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd536225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd5180015c0 con 0x7fd530142690 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: dump: 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:15.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd536225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5180017e0 con 0x7fd530142690 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd52e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd536225640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.160+0000 7fd536225640 4 set_mon_vals no callback set 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd52e7fc640 1 -- 172.21.15.71:0/3548287805 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd518001af0 con 0x7fd530142690 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd52e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd52e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd52e7fc640 10 monclient: dump: 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.162 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd536225640 10 monclient: shutdown 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd536225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:15.163 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd536225640 1 -- 172.21.15.71:0/3548287805 >> v1:172.21.15.81:6789/0 conn(0x7fd530142690 legacy=0x7fd5300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd536225640 1 -- 172.21.15.71:0/3548287805 shutdown_connections 2024-03-29T11:35:15.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.161+0000 7fd536225640 1 -- 172.21.15.71:0/3548287805 >> 172.21.15.71:0/3548287805 conn(0x7fd530066a00 msgr2=0x7fd530066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:15.169 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.162+0000 7fd536225640 1 -- 172.21.15.71:0/3548287805 shutdown_connections 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.162+0000 7fd536225640 1 -- 172.21.15.71:0/3548287805 wait complete. 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.162+0000 7fd536225640 10 monclient: build_initial_monmap 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.162+0000 7fd536225640 10 monclient: monmap: 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:15.170 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 1 librados: starting msgr at 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 1 librados: starting objecter 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 1 Processor -- start 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 1 -- start start 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 1 librados: setting wanted keys 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 1 librados: calling monclient init 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.163+0000 7fd536225640 10 monclient: init 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient(hunting): picked mon.noname-c con 0x7fd5300b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient(hunting): picked mon.noname-a con 0x7fd5301422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient(hunting): picked mon.noname-b con 0x7fd530142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53012e7c0 con 0x7fd5301422b0 2024-03-29T11:35:15.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300ac030 con 0x7fd530142690 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5300ac310 con 0x7fd5300b17d0 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd536225640 10 monclient(hunting): authenticate will time out at 969.318523s 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd52ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd530142690 0x7fd5301de830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:35870/0 (socket says 172.21.15.71:35870) 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd52ffff640 1 -- 172.21.15.71:0/905446213 learned_addr learned my addr 172.21.15.71:0/905446213 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.164+0000 7fd527fff640 10 client.?.objecter ms_handle_connect 0x7fd530142690 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 client.?.objecter ms_handle_connect 0x7fd5301422b0 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131967692 0 0) 0x7fd5300ac030 con 0x7fd530142690 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd510004eb0 con 0x7fd530142690 2024-03-29T11:35:15.172 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 client.?.objecter ms_handle_connect 0x7fd5300b17d0 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188478813 0 0) 0x7fd510004eb0 con 0x7fd530142690 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5300ac030 con 0x7fd530142690 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd528001cb0 con 0x7fd530142690 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): dump: 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:15.173 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290519510 0 0) 0x7fd530151940 con 0x7fd5301422b0 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.165+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd510004eb0 con 0x7fd5301422b0 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1187622595 0 0) 0x7fd5300ac030 con 0x7fd530142690 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 >> v1:172.21.15.71:6790/0 conn(0x7fd5300b17d0 legacy=0x7fd53012e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 >> v1:172.21.15.71:6789/0 conn(0x7fd5301422b0 legacy=0x7fd53012beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 1 monclient: found mon.b 2024-03-29T11:35:15.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53012c5c0 con 0x7fd530142690 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd527fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 10 monclient: _renew_subs 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5301416d0 con 0x7fd530142690 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 10 monclient: _renew_subs 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd530071470 con 0x7fd530142690 2024-03-29T11:35:15.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.166+0000 7fd536225640 1 librados: init done 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd528004b60 con 0x7fd530142690 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd527fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd528004f70 con 0x7fd530142690 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd527fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd527fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd527fff640 10 monclient: dump: 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:15.176 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.167+0000 7fd52dffb640 4 set_mon_vals no callback set 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.168+0000 7fd536225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.168+0000 7fd536225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.168+0000 7fd536225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.168+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd514005080 con 0x7fd530142690 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.169+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fd5280050d0 con 0x7fd530142690 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd52f7fe640 10 monclient: get_auth_request con 0x7fd51006ddc0 auth_method 0 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fd528084de0 con 0x7fd530142690 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd527fff640 10 client.4354.objecter ms_dispatch 0x7fd530000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd527fff640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd527fff640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd527fff640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:35:15.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.170+0000 7fd527fff640 10 client.4354.objecter ms_handle_connect 0x7fd51006ddc0 2024-03-29T11:35:15.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.176+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5280b48b0 con 0x7fd530142690 2024-03-29T11:35:15.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.176+0000 7fd527fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:15.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.176+0000 7fd527fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:15.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.589+0000 7fd536225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.589+0000 7fd536225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.589+0000 7fd536225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.589+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd514005330 con 0x7fd530142690 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.590+0000 7fd527fff640 1 -- 172.21.15.71:0/905446213 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fd5280506c0 con 0x7fd530142690 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.590+0000 7fd527fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.590+0000 7fd527fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:15.588 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:15.589 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 10 librados: watch_flush enter 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 10 librados: watch_flush exit 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 >> v1:172.21.15.71:6816/1596514235 conn(0x7fd51006ddc0 legacy=0x7fd510070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 10 monclient: shutdown 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 >> v1:172.21.15.81:6789/0 conn(0x7fd530142690 legacy=0x7fd5301de830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 shutdown_connections 2024-03-29T11:35:15.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 >> 172.21.15.71:0/905446213 conn(0x7fd530066a00 msgr2=0x7fd53013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:15.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 shutdown_connections 2024-03-29T11:35:15.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 -- 172.21.15.71:0/905446213 wait complete. 2024-03-29T11:35:15.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:15.594+0000 7fd536225640 1 librados: shutdown 2024-03-29T11:35:15.606 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:22.413 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.564+0000 7f1c2b8ff640 10 monclient: get_monmap_and_config 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.564+0000 7f1c2b8ff640 10 monclient: build_initial_monmap 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.564+0000 7f1c2b8ff640 10 monclient: monmap: 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:22.564985+0000 2024-03-29T11:35:22.563 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:22.564985+0000 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.564+0000 7f1c2b8ff640 1 Processor -- start 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.565+0000 7f1c2b8ff640 1 -- start start 2024-03-29T11:35:22.564 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.565+0000 7f1c2b8ff640 10 monclient: init 2024-03-29T11:35:22.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:22.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:22.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient(hunting): picked mon.noname-a con 0x7f1c240a9ac0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:22.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient(hunting): picked mon.noname-c con 0x7f1c2413ee90 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:22.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient(hunting): picked mon.noname-b con 0x7f1c240af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:22.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240afd10 con 0x7f1c240a9ac0 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240af6f0 con 0x7f1c2413ee90 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c240aac80 con 0x7f1c240af900 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c2b8ff640 10 monclient(hunting): authenticate will time out at 976.720258s 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c29e9c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f1c240af900 0x7f1c24149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55566/0 (socket says 172.21.15.71:55566) 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.566+0000 7f1c29e9c640 1 -- 172.21.15.71:0/608876668 learned_addr learned my addr 172.21.15.71:0/608876668 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230690781 0 0) 0x7f1c240aac80 con 0x7f1c240af900 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c10003520 con 0x7f1c240af900 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2291719908 0 0) 0x7f1c240afd10 con 0x7f1c240a9ac0 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c240aac80 con 0x7f1c240a9ac0 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c2969b640 1 -- 172.21.15.71:0/608876668 >> v1:172.21.15.71:6789/0 conn(0x7f1c240a9ac0 legacy=0x7f1c240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c2969b640 1 -- 172.21.15.71:0/608876668 >> v1:172.21.15.71:6789/0 conn(0x7f1c240a9ac0 legacy=0x7f1c240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c2969b640 1 --1- 172.21.15.71:0/608876668 >> v1:172.21.15.71:6789/0 conn(0x7f1c240a9ac0 0x7f1c240adfe0 :-1 s=OPENED pgs=354012 cs=1 l=1).handle_message read tag failed 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614633551 0 0) 0x7f1c240af6f0 con 0x7f1c2413ee90 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.567+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c240afd10 con 0x7f1c2413ee90 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3684517770 0 0) 0x7f1c10003520 con 0x7f1c240af900 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 >> v1:172.21.15.81:6789/0 conn(0x7f1c2413ee90 legacy=0x7f1c2413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 >> v1:172.21.15.71:6789/0 conn(0x7f1c240a9ac0 legacy=0x7f1c240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c2969b640 1 --1- 172.21.15.71:0/608876668 >> v1:172.21.15.71:6789/0 conn(0x7f1c240a9ac0 0x7f1c240adfe0 :-1 s=OPENED pgs=354012 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 1 monclient: found mon.noname-b 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c240aaf60 con 0x7f1c240af900 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:22.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: ms_handle_reset stray mon v1:172.21.15.71:6789/0 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1c14004cb0 con 0x7f1c240af900 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c1bfff640 10 monclient: dump: 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:22.572 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c2b8ff640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.568+0000 7f1c2b8ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c14004e10 con 0x7f1c240af900 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c2b8ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c2b8ff640 4 set_mon_vals no callback set 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c1bfff640 1 -- 172.21.15.71:0/608876668 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1c140013f0 con 0x7f1c240af900 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.569+0000 7f1c1bfff640 10 monclient: dump: 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.573 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 10 monclient: shutdown 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/608876668 >> v1:172.21.15.71:6790/0 conn(0x7f1c240af900 legacy=0x7f1c24149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/608876668 shutdown_connections 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/608876668 >> 172.21.15.71:0/608876668 conn(0x7f1c240b35e0 msgr2=0x7f1c240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:22.574 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/608876668 shutdown_connections 2024-03-29T11:35:22.575 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.570+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/608876668 wait complete. 2024-03-29T11:35:22.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.571+0000 7f1c2b8ff640 10 monclient: build_initial_monmap 2024-03-29T11:35:22.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.571+0000 7f1c2b8ff640 10 monclient: monmap: 2024-03-29T11:35:22.580 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:22.580 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.571+0000 7f1c2b8ff640 1 librados: starting msgr at 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.571+0000 7f1c2b8ff640 1 librados: starting objecter 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.571+0000 7f1c2b8ff640 1 Processor -- start 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.572+0000 7f1c2b8ff640 1 -- start start 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.572+0000 7f1c2b8ff640 1 librados: setting wanted keys 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.572+0000 7f1c2b8ff640 1 librados: calling monclient init 2024-03-29T11:35:22.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.572+0000 7f1c2b8ff640 10 monclient: init 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.572+0000 7f1c2b8ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.572+0000 7f1c2b8ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 10 monclient(hunting): picked mon.noname-c con 0x7f1c240a9ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 10 monclient(hunting): picked mon.noname-b con 0x7f1c240af900 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 10 monclient(hunting): picked mon.noname-a con 0x7f1c2413ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2413f5c0 con 0x7f1c2413ee90 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c24072300 con 0x7f1c240af900 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c24072490 con 0x7f1c240a9ac0 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2969b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f1c240a9ac0 0x7f1c24141fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56684/0 (socket says 172.21.15.71:56684) 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2969b640 1 -- 172.21.15.71:0/168167498 learned_addr learned my addr 172.21.15.71:0/168167498 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c2b8ff640 10 monclient(hunting): authenticate will time out at 976.727424s 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c19ffb640 10 client.?.objecter ms_handle_connect 0x7f1c240af900 2024-03-29T11:35:22.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.573+0000 7f1c19ffb640 10 client.?.objecter ms_handle_connect 0x7f1c240a9ac0 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 client.?.objecter ms_handle_connect 0x7f1c2413ee90 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614839639 0 0) 0x7f1c24072300 con 0x7f1c240af900 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c08004ee0 con 0x7f1c240af900 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358044387 0 0) 0x7f1c2413f5c0 con 0x7f1c2413ee90 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c24072300 con 0x7f1c2413ee90 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090372635 0 0) 0x7f1c24072490 con 0x7f1c240a9ac0 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c2413f5c0 con 0x7f1c240a9ac0 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123615907 0 0) 0x7f1c08004ee0 con 0x7f1c240af900 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c24072490 con 0x7f1c240af900 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1c1c001bf0 con 0x7f1c240af900 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:22.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.574+0000 7f1c19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518767931 0 0) 0x7f1c24072300 con 0x7f1c2413ee90 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c08004ee0 con 0x7f1c2413ee90 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1c14001240 con 0x7f1c2413ee90 2024-03-29T11:35:22.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818696748 0 0) 0x7f1c2413f5c0 con 0x7f1c240a9ac0 2024-03-29T11:35:22.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c24072300 con 0x7f1c240a9ac0 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1c20001860 con 0x7f1c240a9ac0 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 204923221 0 0) 0x7f1c24072490 con 0x7f1c240af900 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 >> v1:172.21.15.71:6790/0 conn(0x7f1c240a9ac0 legacy=0x7f1c24141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 >> v1:172.21.15.71:6789/0 conn(0x7f1c2413ee90 legacy=0x7f1c241d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 monclient: found mon.b 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c241ca220 con 0x7f1c240af900 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c2b8ff640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c2b8ff640 10 monclient: _renew_subs 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c2b8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.575+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2413b250 con 0x7f1c240af900 2024-03-29T11:35:22.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.576+0000 7f1c2b8ff640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.576+0000 7f1c2b8ff640 10 monclient: _renew_subs 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.576+0000 7f1c2b8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.576+0000 7f1c2b8ff640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c241d7dc0 con 0x7f1c240af900 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.576+0000 7f1c2b8ff640 1 librados: init done 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c1c003970 con 0x7f1c240af900 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1c1c003c80 con 0x7f1c240af900 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c1b7fe640 4 set_mon_vals no callback set 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c19ffb640 10 monclient: dump: 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:22.589 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.577+0000 7f1c0f7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.578+0000 7f1c0f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.578+0000 7f1c0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f1c1c019c50 con 0x7f1c240af900 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.578+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc005080 con 0x7f1c240af900 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f1c1c083c30 con 0x7f1c240af900 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c19ffb640 10 client.4360.objecter ms_dispatch 0x7f1c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:22.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c19ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c19ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c19ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.579+0000 7f1c2969b640 10 monclient: get_auth_request con 0x7f1c0806e1a0 auth_method 0 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.580+0000 7f1c19ffb640 10 client.4360.objecter ms_handle_connect 0x7f1c0806e1a0 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.587+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c1c04f510 con 0x7f1c240af900 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.587+0000 7f1c19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:22.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.588+0000 7f1c19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:22.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.928+0000 7f1c0f7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:22.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.928+0000 7f1c0f7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:22.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.928+0000 7f1c0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:22.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.928+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bfc005330 con 0x7f1c240af900 2024-03-29T11:35:22.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.929+0000 7f1c19ffb640 1 -- 172.21.15.71:0/168167498 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f1c1c051440 con 0x7f1c240af900 2024-03-29T11:35:22.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.929+0000 7f1c19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:22.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.929+0000 7f1c19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:22.927 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:22.927 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 10 librados: watch_flush enter 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 10 librados: watch_flush exit 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 >> v1:172.21.15.71:6816/1596514235 conn(0x7f1c0806e1a0 legacy=0x7f1c08070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 10 monclient: shutdown 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 >> v1:172.21.15.81:6789/0 conn(0x7f1c240af900 legacy=0x7f1c2413ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 shutdown_connections 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.933+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 >> 172.21.15.71:0/168167498 conn(0x7f1c240b35e0 msgr2=0x7f1c240adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.934+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 shutdown_connections 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.934+0000 7f1c0f7fe640 1 -- 172.21.15.71:0/168167498 wait complete. 2024-03-29T11:35:22.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:22.934+0000 7f1c0f7fe640 1 librados: shutdown 2024-03-29T11:35:22.946 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:29.752 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:29.891 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.892+0000 7ff870bcf640 10 monclient: get_monmap_and_config 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.892+0000 7ff870bcf640 10 monclient: build_initial_monmap 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.892+0000 7ff870bcf640 10 monclient: monmap: 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:29.893275+0000 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:29.893275+0000 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:29.892 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.894+0000 7ff870bcf640 1 Processor -- start 2024-03-29T11:35:29.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.894+0000 7ff870bcf640 1 -- start start 2024-03-29T11:35:29.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.894+0000 7ff870bcf640 10 monclient: init 2024-03-29T11:35:29.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:29.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:29.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff86c1457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:29.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff86c13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:29.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff86c0a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:29.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c0aaef0 con 0x7ff86c1457f0 2024-03-29T11:35:29.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c145bd0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c145eb0 con 0x7ff86c13f650 2024-03-29T11:35:29.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:29.895 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.896+0000 7ff870bcf640 10 monclient(hunting): authenticate will time out at 984.050300s 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86ad9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff86c0a9d70 0x7ff86c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:57590/0 (socket says 172.21.15.71:57590) 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86ad9d640 1 -- 172.21.15.71:0/4096727405 learned_addr learned my addr 172.21.15.71:0/4096727405 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379871073 0 0) 0x7ff86c145bd0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff854003520 con 0x7ff86c0a9d70 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.898+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3095974739 0 0) 0x7ff854003520 con 0x7ff86c0a9d70 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 >> v1:172.21.15.71:6790/0 conn(0x7ff86c13f650 legacy=0x7ff86c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:29.897 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 >> v1:172.21.15.71:6789/0 conn(0x7ff86c1457f0 legacy=0x7ff86c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:29.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:29.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 1 monclient: found mon.noname-c 2024-03-29T11:35:29.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:29.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:35:29.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c146340 con 0x7ff86c0a9d70 2024-03-29T11:35:29.899 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 10 monclient: _finish_auth 0 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff870bcf640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff870bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff860001690 con 0x7ff86c0a9d70 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.899+0000 7ff86959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 10 monclient: dump: 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:29.900 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff870bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff860001920 con 0x7ff86c0a9d70 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 1 -- 172.21.15.71:0/4096727405 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff860001cb0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff86959a640 10 monclient: dump: 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:29.901 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff870bcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.900+0000 7ff870bcf640 4 set_mon_vals no callback set 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.901+0000 7ff870bcf640 10 monclient: shutdown 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.901+0000 7ff870bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:29.902 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.901+0000 7ff870bcf640 1 -- 172.21.15.71:0/4096727405 >> v1:172.21.15.81:6789/0 conn(0x7ff86c0a9d70 legacy=0x7ff86c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.901+0000 7ff870bcf640 1 -- 172.21.15.71:0/4096727405 shutdown_connections 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.901+0000 7ff870bcf640 1 -- 172.21.15.71:0/4096727405 >> 172.21.15.71:0/4096727405 conn(0x7ff86c066a00 msgr2=0x7ff86c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.902+0000 7ff870bcf640 1 -- 172.21.15.71:0/4096727405 shutdown_connections 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.902+0000 7ff870bcf640 1 -- 172.21.15.71:0/4096727405 wait complete. 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.903+0000 7ff870bcf640 10 monclient: build_initial_monmap 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.903+0000 7ff870bcf640 10 monclient: monmap: 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:29.904 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.903+0000 7ff870bcf640 1 librados: starting msgr at 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.904+0000 7ff870bcf640 1 librados: starting objecter 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.904+0000 7ff870bcf640 1 Processor -- start 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.904+0000 7ff870bcf640 1 -- start start 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.904+0000 7ff870bcf640 1 librados: setting wanted keys 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.904+0000 7ff870bcf640 1 librados: calling monclient init 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.904+0000 7ff870bcf640 10 monclient: init 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.905+0000 7ff870bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:29.905 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.905+0000 7ff870bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.905+0000 7ff870bcf640 10 monclient(hunting): picked mon.noname-a con 0x7ff86c0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 10 monclient(hunting): picked mon.noname-b con 0x7ff86c13f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 10 monclient(hunting): picked mon.noname-c con 0x7ff86c1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c133d70 con 0x7ff86c0a9d70 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c135f30 con 0x7ff86c13f650 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86c1317c0 con 0x7ff86c1457f0 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.906+0000 7ff870bcf640 10 monclient(hunting): authenticate will time out at 984.061044s 2024-03-29T11:35:29.906 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff869d9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff86c13f650 0x7ff86c1309a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:57960/0 (socket says 172.21.15.71:57960) 2024-03-29T11:35:29.907 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff869d9b640 1 -- 172.21.15.71:0/1568601578 learned_addr learned my addr 172.21.15.71:0/1568601578 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:29.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 client.?.objecter ms_handle_connect 0x7ff86c13f650 2024-03-29T11:35:29.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:29.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231797395 0 0) 0x7ff86c135f30 con 0x7ff86c13f650 2024-03-29T11:35:29.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:29.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:29.908 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff84c004eb0 con 0x7ff86c13f650 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 client.?.objecter ms_handle_connect 0x7ff86c0a9d70 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.908+0000 7ff85affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445986543 0 0) 0x7ff86c133d70 con 0x7ff86c0a9d70 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff86c135f30 con 0x7ff86c0a9d70 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3495333199 0 0) 0x7ff84c004eb0 con 0x7ff86c13f650 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff86c133d70 con 0x7ff86c13f650 2024-03-29T11:35:29.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff85c001960 con 0x7ff86c13f650 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): dump: 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1497371461 0 0) 0x7ff86c135f30 con 0x7ff86c0a9d70 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff84c004eb0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff864001b10 con 0x7ff86c0a9d70 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient(hunting): dump: 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:29.911 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 222794769 0 0) 0x7ff84c004eb0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 >> v1:172.21.15.71:6790/0 conn(0x7ff86c1457f0 legacy=0x7ff86c1310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 >> v1:172.21.15.81:6789/0 conn(0x7ff86c13f650 legacy=0x7ff86c1309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 monclient: found mon.a 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c134200 con 0x7ff86c0a9d70 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff85affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff870bcf640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff870bcf640 10 monclient: _renew_subs 2024-03-29T11:35:29.912 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff870bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:29.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.909+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86c1365c0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.910+0000 7ff870bcf640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:29.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.910+0000 7ff870bcf640 10 monclient: _renew_subs 2024-03-29T11:35:29.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.910+0000 7ff870bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:29.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.910+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86c1dade0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.910+0000 7ff870bcf640 1 librados: init done 2024-03-29T11:35:29.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.911+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff864002020 con 0x7ff86c0a9d70 2024-03-29T11:35:29.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.911+0000 7ff85affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:29.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.911+0000 7ff868d99640 4 set_mon_vals no callback set 2024-03-29T11:35:29.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.911+0000 7ff870bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff864003b10 con 0x7ff86c0a9d70 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff870bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff870bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff850005080 con 0x7ff86c0a9d70 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff85affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff85affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff85affd640 10 monclient: dump: 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:29.915 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.912+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7ff864019ae0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.913+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7ff864083b50 con 0x7ff86c0a9d70 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.913+0000 7ff85affd640 10 client.4353.objecter ms_dispatch 0x7ff86c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.913+0000 7ff85affd640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.913+0000 7ff85affd640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.913+0000 7ff85affd640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.915+0000 7ff869d9b640 10 monclient: get_auth_request con 0x7ff84c06de30 auth_method 0 2024-03-29T11:35:29.916 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.915+0000 7ff85affd640 10 client.4353.objecter ms_handle_connect 0x7ff84c06de30 2024-03-29T11:35:29.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.922+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff86404f2e0 con 0x7ff86c0a9d70 2024-03-29T11:35:29.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.922+0000 7ff85affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:29.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:29.922+0000 7ff85affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:30.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.253+0000 7ff870bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:30.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.253+0000 7ff870bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:30.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.253+0000 7ff870bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:30.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.253+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff850005330 con 0x7ff86c0a9d70 2024-03-29T11:35:30.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.254+0000 7ff85affd640 1 -- 172.21.15.71:0/1568601578 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7ff8640513e0 con 0x7ff86c0a9d70 2024-03-29T11:35:30.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.254+0000 7ff85affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:30.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.254+0000 7ff85affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:30.254 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:30.254 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:30.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.259+0000 7ff870bcf640 10 librados: watch_flush enter 2024-03-29T11:35:30.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.261+0000 7ff870bcf640 10 librados: watch_flush exit 2024-03-29T11:35:30.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.261+0000 7ff870bcf640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-29T11:35:30.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.261+0000 7ff870bcf640 10 client.4353.objecter successfully canceled tick 2024-03-29T11:35:30.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 >> v1:172.21.15.71:6816/1596514235 conn(0x7ff84c06de30 legacy=0x7ff84c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 10 monclient: shutdown 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 >> v1:172.21.15.71:6789/0 conn(0x7ff86c0a9d70 legacy=0x7ff86c1da4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 shutdown_connections 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 >> 172.21.15.71:0/1568601578 conn(0x7ff86c066a00 msgr2=0x7ff86c0b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 shutdown_connections 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.262+0000 7ff870bcf640 1 -- 172.21.15.71:0/1568601578 wait complete. 2024-03-29T11:35:30.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:30.263+0000 7ff870bcf640 1 librados: shutdown 2024-03-29T11:35:30.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:37.083 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.228+0000 7fb2f32b3640 10 monclient: get_monmap_and_config 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.228+0000 7fb2f32b3640 10 monclient: build_initial_monmap 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.228+0000 7fb2f32b3640 10 monclient: monmap: 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:37.229639+0000 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:37.229639+0000 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.227 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:37.228 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:37.228 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:37.228 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.230+0000 7fb2f32b3640 1 Processor -- start 2024-03-29T11:35:37.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.231+0000 7fb2f32b3640 1 -- start start 2024-03-29T11:35:37.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.231+0000 7fb2f32b3640 10 monclient: init 2024-03-29T11:35:37.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.231+0000 7fb2f32b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:37.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.231+0000 7fb2f32b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.231+0000 7fb2f32b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2ec0b4330 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.231+0000 7fb2f32b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb2ec1470b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f32b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb2ec1439b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f32b3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec06f2a0 con 0x7fb2ec1470b0 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f32b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec146b90 con 0x7fb2ec1439b0 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f32b3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec0b2ca0 con 0x7fb2ec0b4330 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f32b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f32b3640 10 monclient(hunting): authenticate will time out at 991.386096s 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f104f640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb2ec0b4330 0x7fb2ec0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34050/0 (socket says 172.21.15.71:34050) 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2f104f640 1 -- 172.21.15.71:0/3529486277 learned_addr learned my addr 172.21.15.71:0/3529486277 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278586554 0 0) 0x7fb2ec0b2ca0 con 0x7fb2ec0b4330 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:35:37.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d8003520 con 0x7fb2ec0b4330 2024-03-29T11:35:37.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1653189559 0 0) 0x7fb2d8003520 con 0x7fb2ec0b4330 2024-03-29T11:35:37.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.232+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 >> v1:172.21.15.81:6789/0 conn(0x7fb2ec1439b0 legacy=0x7fb2ec151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:37.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.235+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 >> v1:172.21.15.71:6789/0 conn(0x7fb2ec1470b0 legacy=0x7fb2ec0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 1 monclient: found mon.noname-b 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec06e0c0 con 0x7fb2ec0b4330 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:37.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2f32b3640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2f32b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb2e8001670 con 0x7fb2ec0b4330 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2e3fff640 10 monclient: dump: 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.240 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.238+0000 7fb2f32b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.240+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2e80018d0 con 0x7fb2ec0b4330 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.240+0000 7fb2e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.240+0000 7fb2e3fff640 1 -- 172.21.15.71:0/3529486277 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb2e8001d20 con 0x7fb2ec0b4330 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.240+0000 7fb2e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.240+0000 7fb2e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.240+0000 7fb2e3fff640 10 monclient: dump: 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.241 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.242 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:37.242 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:37.242 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:37.242 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.241+0000 7fb2f32b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:37.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.241+0000 7fb2f32b3640 4 set_mon_vals no callback set 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.241+0000 7fb2f32b3640 10 monclient: shutdown 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.241+0000 7fb2f32b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.241+0000 7fb2f32b3640 1 -- 172.21.15.71:0/3529486277 >> v1:172.21.15.71:6790/0 conn(0x7fb2ec0b4330 legacy=0x7fb2ec0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.242+0000 7fb2f32b3640 1 -- 172.21.15.71:0/3529486277 shutdown_connections 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.242+0000 7fb2f32b3640 1 -- 172.21.15.71:0/3529486277 >> 172.21.15.71:0/3529486277 conn(0x7fb2ec066a00 msgr2=0x7fb2ec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.244+0000 7fb2f32b3640 1 -- 172.21.15.71:0/3529486277 shutdown_connections 2024-03-29T11:35:37.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.245+0000 7fb2f32b3640 1 -- 172.21.15.71:0/3529486277 wait complete. 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.245+0000 7fb2f32b3640 10 monclient: build_initial_monmap 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.245+0000 7fb2f32b3640 10 monclient: monmap: 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.244 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 1 librados: starting msgr at 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 1 librados: starting objecter 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 1 Processor -- start 2024-03-29T11:35:37.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 1 -- start start 2024-03-29T11:35:37.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 1 librados: setting wanted keys 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 1 librados: calling monclient init 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.246+0000 7fb2f32b3640 10 monclient: init 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2ec0b4330 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb2ec1439b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb2ec1470b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec142640 con 0x7fb2ec1439b0 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec13b740 con 0x7fb2ec0b4330 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec143300 con 0x7fb2ec1470b0 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.247+0000 7fb2f32b3640 10 monclient(hunting): authenticate will time out at 991.401535s 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.248+0000 7fb2f1850640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb2ec1470b0 0x7fb2ec13b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39236/0 (socket says 172.21.15.71:39236) 2024-03-29T11:35:37.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.248+0000 7fb2f1850640 1 -- 172.21.15.71:0/2585301347 learned_addr learned my addr 172.21.15.71:0/2585301347 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:37.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2ec1470b0 2024-03-29T11:35:37.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:37.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634306151 0 0) 0x7fb2ec143300 con 0x7fb2ec1470b0 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.249+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0004eb0 con 0x7fb2ec1470b0 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276647381 0 0) 0x7fb2d0004eb0 con 0x7fb2ec1470b0 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2ec143300 con 0x7fb2ec1470b0 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb2dc0017f0 con 0x7fb2ec1470b0 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:37.249 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.250+0000 7fb2e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:37.251 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2ec0b4330 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374183464 0 0) 0x7fb2ec13b740 con 0x7fb2ec0b4330 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0004eb0 con 0x7fb2ec0b4330 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 client.?.objecter ms_handle_connect 0x7fb2ec1439b0 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.251+0000 7fb2e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1620119589 0 0) 0x7fb2d0004eb0 con 0x7fb2ec0b4330 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2ec13b740 con 0x7fb2ec0b4330 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb2e8001630 con 0x7fb2ec0b4330 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1485640432 0 0) 0x7fb2ec13b740 con 0x7fb2ec0b4330 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 >> v1:172.21.15.71:6790/0 conn(0x7fb2ec1470b0 legacy=0x7fb2ec13b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 >> v1:172.21.15.71:6789/0 conn(0x7fb2ec1439b0 legacy=0x7fb2ec13a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 monclient: found mon.b 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec0b1540 con 0x7fb2ec0b4330 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:37.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.252+0000 7fb2e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 10 monclient: _renew_subs 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec0b1790 con 0x7fb2ec0b4330 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 10 monclient: _renew_subs 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec1d6270 con 0x7fb2ec0b4330 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.253+0000 7fb2f32b3640 1 librados: init done 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2e8001210 con 0x7fb2ec0b4330 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2f32b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2f32b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:37.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2f32b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec075f10 con 0x7fb2ec0b4330 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb2e8015b30 con 0x7fb2ec0b4330 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.255+0000 7fb2e1ffb640 10 monclient: dump: 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:37.257 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:37.258 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:37.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.256+0000 7fb2e37fe640 4 set_mon_vals no callback set 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.258+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fb2e802bb00 con 0x7fb2ec0b4330 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.259+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fb2e8095870 con 0x7fb2ec0b4330 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.259+0000 7fb2e1ffb640 10 client.4369.objecter ms_dispatch 0x7fb2ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.259+0000 7fb2e1ffb640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.259+0000 7fb2e1ffb640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.259+0000 7fb2e1ffb640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.260+0000 7fb2f084e640 10 monclient: get_auth_request con 0x7fb2d006dfb0 auth_method 0 2024-03-29T11:35:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.261+0000 7fb2e1ffb640 10 client.4369.objecter ms_handle_connect 0x7fb2d006dfb0 2024-03-29T11:35:37.261 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.263+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2e8061150 con 0x7fb2ec0b4330 2024-03-29T11:35:37.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.264+0000 7fb2e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:37.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.264+0000 7fb2e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:37.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.588+0000 7fb2f32b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:37.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.588+0000 7fb2f32b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:37.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.588+0000 7fb2f32b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:35:37.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.588+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2ec074f10 con 0x7fb2ec0b4330 2024-03-29T11:35:37.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.590+0000 7fb2e1ffb640 1 -- 172.21.15.71:0/2585301347 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fb2e8063080 con 0x7fb2ec0b4330 2024-03-29T11:35:37.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.590+0000 7fb2e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:37.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.590+0000 7fb2e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:37.589 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:37.589 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:37.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 10 librados: watch_flush enter 2024-03-29T11:35:37.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 10 librados: watch_flush exit 2024-03-29T11:35:37.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T11:35:37.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 10 client.4369.objecter successfully canceled tick 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 >> v1:172.21.15.71:6816/1596514235 conn(0x7fb2d006dfb0 legacy=0x7fb2d0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 10 monclient: shutdown 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.594+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 >> v1:172.21.15.81:6789/0 conn(0x7fb2ec0b4330 legacy=0x7fb2ec142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.595+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 shutdown_connections 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.595+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 >> 172.21.15.71:0/2585301347 conn(0x7fb2ec066a00 msgr2=0x7fb2ec0b71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:37.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.596+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 shutdown_connections 2024-03-29T11:35:37.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.596+0000 7fb2f32b3640 1 -- 172.21.15.71:0/2585301347 wait complete. 2024-03-29T11:35:37.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:37.596+0000 7fb2f32b3640 1 librados: shutdown 2024-03-29T11:35:37.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:44.418 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.571+0000 7f556404a640 10 monclient: get_monmap_and_config 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.571+0000 7f556404a640 10 monclient: build_initial_monmap 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.571+0000 7f556404a640 10 monclient: monmap: 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:44.573641+0000 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:44.573641+0000 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:44.571 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:44.572 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:44.572 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:44.572 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:44.572 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:44.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.572+0000 7f556404a640 1 Processor -- start 2024-03-29T11:35:44.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.573+0000 7f556404a640 1 -- start start 2024-03-29T11:35:44.573 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.573+0000 7f556404a640 10 monclient: init 2024-03-29T11:35:44.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:44.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:44.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient(hunting): picked mon.noname-a con 0x7f555c0b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:44.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient(hunting): picked mon.noname-b con 0x7f555c071eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:44.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient(hunting): picked mon.noname-c con 0x7f555c143d10 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:44.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c06ef80 con 0x7f555c0b1f60 2024-03-29T11:35:44.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c06f260 con 0x7f555c143d10 2024-03-29T11:35:44.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c0b40c0 con 0x7f555c071eb0 2024-03-29T11:35:44.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:44.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.574+0000 7f556404a640 10 monclient(hunting): authenticate will time out at 998.729305s 2024-03-29T11:35:44.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.579+0000 7f55615e5640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f555c071eb0 0x7f555c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42609/0 (socket says 172.21.15.71:42609) 2024-03-29T11:35:44.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.579+0000 7f55615e5640 1 -- 172.21.15.71:0/2859165335 learned_addr learned my addr 172.21.15.71:0/2859165335 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.580+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104676827 0 0) 0x7f555c0b40c0 con 0x7f555c071eb0 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.580+0000 7f5560de4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.580+0000 7f5560de4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.580+0000 7f5560de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.580+0000 7f5560de4640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.580+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5548003520 con 0x7f555c071eb0 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1126255852 0 0) 0x7f5548003520 con 0x7f555c071eb0 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 >> v1:172.21.15.81:6789/0 conn(0x7f555c143d10 legacy=0x7f555c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 >> v1:172.21.15.71:6789/0 conn(0x7f555c0b1f60 legacy=0x7f555c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:44.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: _finish_hunting 0 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 1 monclient: found mon.noname-b 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c155d70 con 0x7f555c071eb0 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: _finish_auth 0 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: _check_auth_tickets 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5554001a10 con 0x7f555c071eb0 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.581+0000 7f5560de4640 10 monclient: dump: 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.583 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.583+0000 7f556404a640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:35:44.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.583+0000 7f556404a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:44.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5554001be0 con 0x7f555c071eb0 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f5560de4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f5560de4640 1 -- 172.21.15.71:0/2859165335 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5554001ff0 con 0x7f555c071eb0 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f5560de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f5560de4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f5560de4640 10 monclient: dump: 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:44.587 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:44.588 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:44.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f556404a640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:44.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.587+0000 7f556404a640 4 set_mon_vals no callback set 2024-03-29T11:35:44.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.588+0000 7f556404a640 10 monclient: shutdown 2024-03-29T11:35:44.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.588+0000 7f556404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:44.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.588+0000 7f556404a640 1 -- 172.21.15.71:0/2859165335 >> v1:172.21.15.71:6790/0 conn(0x7f555c071eb0 legacy=0x7f555c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.590+0000 7f556404a640 1 -- 172.21.15.71:0/2859165335 shutdown_connections 2024-03-29T11:35:44.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.590+0000 7f556404a640 1 -- 172.21.15.71:0/2859165335 >> 172.21.15.71:0/2859165335 conn(0x7f555c066a00 msgr2=0x7f555c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:44.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.592+0000 7f556404a640 1 -- 172.21.15.71:0/2859165335 shutdown_connections 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.592+0000 7f556404a640 1 -- 172.21.15.71:0/2859165335 wait complete. 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.593+0000 7f556404a640 10 monclient: build_initial_monmap 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.593+0000 7f556404a640 10 monclient: monmap: 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:44.593 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.593+0000 7f556404a640 1 librados: starting msgr at 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.593+0000 7f556404a640 1 librados: starting objecter 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 1 Processor -- start 2024-03-29T11:35:44.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 1 -- start start 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 1 librados: setting wanted keys 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 1 librados: calling monclient init 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 10 monclient: init 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.594+0000 7f556404a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 10 monclient(hunting): picked mon.noname-c con 0x7f555c0b1f60 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 10 monclient(hunting): picked mon.noname-b con 0x7f555c143d10 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:44.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 10 monclient(hunting): picked mon.noname-a con 0x7f555c1483f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:44.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c14ab60 con 0x7f555c1483f0 2024-03-29T11:35:44.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c14ae40 con 0x7f555c143d10 2024-03-29T11:35:44.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f555c147ef0 con 0x7f555c0b1f60 2024-03-29T11:35:44.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:44.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.595+0000 7f556404a640 10 monclient(hunting): authenticate will time out at 998.750108s 2024-03-29T11:35:44.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f5561de6640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f555c0b1f60 0x7f555c13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42747/0 (socket says 172.21.15.71:42747) 2024-03-29T11:35:44.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f5561de6640 1 -- 172.21.15.71:0/1348433293 learned_addr learned my addr 172.21.15.71:0/1348433293 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:44.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 10 client.?.objecter ms_handle_connect 0x7f555c1483f0 2024-03-29T11:35:44.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:44.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222756347 0 0) 0x7f555c14ab60 con 0x7f555c1483f0 2024-03-29T11:35:44.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:35:44.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.601+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5540004eb0 con 0x7f555c1483f0 2024-03-29T11:35:44.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1260906292 0 0) 0x7f5540004eb0 con 0x7f555c1483f0 2024-03-29T11:35:44.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f555c14ab60 con 0x7f555c1483f0 2024-03-29T11:35:44.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f554c001740 con 0x7f555c1483f0 2024-03-29T11:35:44.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:44.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 10 monclient(hunting): dump: 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522431239 0 0) 0x7f555c14ab60 con 0x7f555c1483f0 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 >> v1:172.21.15.71:6790/0 conn(0x7f555c0b1f60 legacy=0x7f555c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.602+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 >> v1:172.21.15.81:6789/0 conn(0x7f555c143d10 legacy=0x7f555c149210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 1 monclient: found mon.a 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c13fb10 con 0x7f555c1483f0 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f554c0018a0 con 0x7f555c1483f0 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f554c0037b0 con 0x7f555c1483f0 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55527fc640 10 monclient: dump: 2024-03-29T11:35:44.608 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.604+0000 7f55537fe640 4 set_mon_vals no callback set 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 10 monclient: _renew_subs 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:44.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555c14a030 con 0x7f555c1483f0 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 10 monclient: _renew_subs 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f555c070890 con 0x7f555c1483f0 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.605+0000 7f556404a640 1 librados: init done 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.606+0000 7f556404a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.606+0000 7f556404a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:44.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.606+0000 7f556404a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.606+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f555c148080 con 0x7f555c1483f0 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.610+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f554c003910 con 0x7f555c1483f0 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.611+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f554c0837b0 con 0x7f555c1483f0 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.611+0000 7f55527fc640 10 client.4359.objecter ms_dispatch 0x7f555c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.611+0000 7f55527fc640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.611+0000 7f55527fc640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:44.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.612+0000 7f55527fc640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:35:44.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.613+0000 7f5561de6640 10 monclient: get_auth_request con 0x7f554006dc00 auth_method 0 2024-03-29T11:35:44.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.615+0000 7f55527fc640 10 client.4359.objecter ms_handle_connect 0x7f554006dc00 2024-03-29T11:35:44.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.620+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f554c04f0f0 con 0x7f555c1483f0 2024-03-29T11:35:44.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.620+0000 7f55527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:44.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.620+0000 7f55527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:44.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.957+0000 7f556404a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:44.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.957+0000 7f556404a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:44.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.957+0000 7f556404a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:44.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.957+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f555c075f10 con 0x7f555c1483f0 2024-03-29T11:35:44.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.959+0000 7f55527fc640 1 -- 172.21.15.71:0/1348433293 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f554c051020 con 0x7f555c1483f0 2024-03-29T11:35:44.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.959+0000 7f55527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:44.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.959+0000 7f55527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:44.960 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:44.960 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 10 librados: watch_flush enter 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 10 librados: watch_flush exit 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 >> v1:172.21.15.71:6816/1596514235 conn(0x7f554006dc00 legacy=0x7f55400700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 10 monclient: shutdown 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:44.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.962+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 >> v1:172.21.15.71:6789/0 conn(0x7f555c1483f0 legacy=0x7f555c149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.965+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 shutdown_connections 2024-03-29T11:35:44.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.965+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 >> 172.21.15.71:0/1348433293 conn(0x7f555c066a00 msgr2=0x7f555c0b7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:44.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.966+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 shutdown_connections 2024-03-29T11:35:44.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.967+0000 7f556404a640 1 -- 172.21.15.71:0/1348433293 wait complete. 2024-03-29T11:35:44.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:44.967+0000 7f556404a640 1 librados: shutdown 2024-03-29T11:35:44.986 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:51.793 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:51.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.946+0000 7f86f830e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:51.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.946+0000 7f86f830e640 10 monclient: build_initial_monmap 2024-03-29T11:35:51.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.947+0000 7f86f830e640 10 monclient: monmap: 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:51.947915+0000 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:51.947915+0000 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:51.948 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.951+0000 7f86f830e640 1 Processor -- start 2024-03-29T11:35:51.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.953+0000 7f86f830e640 1 -- start start 2024-03-29T11:35:51.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.953+0000 7f86f830e640 10 monclient: init 2024-03-29T11:35:51.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.954+0000 7f86f830e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:51.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.954+0000 7f86f830e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.954+0000 7f86f830e640 10 monclient(hunting): picked mon.noname-c con 0x7f86f00ae250 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 10 monclient(hunting): picked mon.noname-a con 0x7f86f013f970 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 10 monclient(hunting): picked mon.noname-b con 0x7f86f0142800 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f00afe00 con 0x7f86f013f970 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f0142c10 con 0x7f86f00ae250 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f00afb20 con 0x7f86f0142800 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f830e640 10 monclient(hunting): authenticate will time out at 1006.112714s 2024-03-29T11:35:51.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f60aa640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f86f00ae250 0x7f86f013f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:35251/0 (socket says 172.21.15.71:35251) 2024-03-29T11:35:51.958 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.958+0000 7f86f60aa640 1 -- 172.21.15.71:0/3409361206 learned_addr learned my addr 172.21.15.71:0/3409361206 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:51.961 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.961+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013674764 0 0) 0x7f86f0142c10 con 0x7f86f00ae250 2024-03-29T11:35:51.961 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.961+0000 7f86f50a8640 10 monclient(hunting): _negotiate 2024-03-29T11:35:51.961 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.961+0000 7f86f50a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:51.961 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.961+0000 7f86f50a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.961+0000 7f86f50a8640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.961+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc003520 con 0x7f86f00ae250 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752504948 0 0) 0x7f86f00afe00 con 0x7f86f013f970 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient(hunting): _negotiate 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f0142c10 con 0x7f86f013f970 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573408481 0 0) 0x7f86f0142c10 con 0x7f86f013f970 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 >> v1:172.21.15.71:6790/0 conn(0x7f86f0142800 legacy=0x7f86f014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 >> v1:172.21.15.81:6789/0 conn(0x7f86f00ae250 legacy=0x7f86f013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: _finish_hunting 0 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 monclient: found mon.noname-a 2024-03-29T11:35:51.962 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f00aab80 con 0x7f86f013f970 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: _finish_auth 0 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: _check_auth_tickets 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f86e0001670 con 0x7f86f013f970 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f50a8640 10 monclient: dump: 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:51.963 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f830e640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.962+0000 7f86f830e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86e00017d0 con 0x7f86f013f970 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f50a8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f50a8640 1 -- 172.21.15.71:0/3409361206 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f86e0001c20 con 0x7f86f013f970 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f50a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f50a8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f50a8640 10 monclient: dump: 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.964 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f830e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:51.965 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.963+0000 7f86f830e640 4 set_mon_vals no callback set 2024-03-29T11:35:51.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.964+0000 7f86f830e640 10 monclient: shutdown 2024-03-29T11:35:51.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.964+0000 7f86f830e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:51.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.964+0000 7f86f830e640 1 -- 172.21.15.71:0/3409361206 >> v1:172.21.15.71:6789/0 conn(0x7f86f013f970 legacy=0x7f86f01420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:51.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.966+0000 7f86f830e640 1 -- 172.21.15.71:0/3409361206 shutdown_connections 2024-03-29T11:35:51.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.966+0000 7f86f830e640 1 -- 172.21.15.71:0/3409361206 >> 172.21.15.71:0/3409361206 conn(0x7f86f00b35e0 msgr2=0x7f86f00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:51.966 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.967+0000 7f86f830e640 1 -- 172.21.15.71:0/3409361206 shutdown_connections 2024-03-29T11:35:51.968 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.970+0000 7f86f830e640 1 -- 172.21.15.71:0/3409361206 wait complete. 2024-03-29T11:35:51.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.970+0000 7f86f830e640 10 monclient: build_initial_monmap 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.970+0000 7f86f830e640 10 monclient: monmap: 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.971+0000 7f86f830e640 1 librados: starting msgr at 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.971+0000 7f86f830e640 1 librados: starting objecter 2024-03-29T11:35:51.970 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.971+0000 7f86f830e640 1 Processor -- start 2024-03-29T11:35:51.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.972+0000 7f86f830e640 1 -- start start 2024-03-29T11:35:51.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.972+0000 7f86f830e640 1 librados: setting wanted keys 2024-03-29T11:35:51.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.972+0000 7f86f830e640 1 librados: calling monclient init 2024-03-29T11:35:51.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.972+0000 7f86f830e640 10 monclient: init 2024-03-29T11:35:51.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.973+0000 7f86f830e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:51.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:51.972 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 10 monclient(hunting): picked mon.noname-c con 0x7f86f00ae250 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:51.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 10 monclient(hunting): picked mon.noname-b con 0x7f86f013f970 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:51.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 10 monclient(hunting): picked mon.noname-a con 0x7f86f0142800 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:51.973 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f0145c90 con 0x7f86f0142800 2024-03-29T11:35:51.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f01d7620 con 0x7f86f013f970 2024-03-29T11:35:51.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f0143930 con 0x7f86f00ae250 2024-03-29T11:35:51.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:51.974 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.974+0000 7f86f830e640 10 monclient(hunting): authenticate will time out at 1006.128920s 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.978+0000 7f86f58a9640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f86f013f970 0x7f86f00717d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:35357/0 (socket says 172.21.15.71:35357) 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.978+0000 7f86f58a9640 1 -- 172.21.15.71:0/487948037 learned_addr learned my addr 172.21.15.71:0/487948037 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 client.?.objecter ms_handle_connect 0x7f86f013f970 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 client.?.objecter ms_handle_connect 0x7f86f0142800 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:51.978 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986232608 0 0) 0x7f86f01d7620 con 0x7f86f013f970 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d4004ee0 con 0x7f86f013f970 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070355103 0 0) 0x7f86f0145c90 con 0x7f86f0142800 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.979+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f01d7620 con 0x7f86f0142800 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779017210 0 0) 0x7f86d4004ee0 con 0x7f86f013f970 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d40054c0 con 0x7f86f013f970 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f86e00035f0 con 0x7f86f013f970 2024-03-29T11:35:51.979 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 client.?.objecter ms_handle_connect 0x7f86f00ae250 2024-03-29T11:35:51.980 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:51.981 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120379855 0 0) 0x7f86f01d7620 con 0x7f86f0142800 2024-03-29T11:35:51.981 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86d4005b90 con 0x7f86f0142800 2024-03-29T11:35:51.981 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f86ec001640 con 0x7f86f0142800 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3478348217 0 0) 0x7f86d4005b90 con 0x7f86f0142800 2024-03-29T11:35:51.982 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 >> v1:172.21.15.71:6790/0 conn(0x7f86f00ae250 legacy=0x7f86f01d6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 >> v1:172.21.15.81:6789/0 conn(0x7f86f013f970 legacy=0x7f86f00717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 monclient: found mon.a 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f0146040 con 0x7f86f0142800 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.980+0000 7f86e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 10 monclient: _renew_subs 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86ec001b70 con 0x7f86f0142800 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:51.983 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f0146290 con 0x7f86f0142800 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f86ec001640 con 0x7f86f0142800 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86e6ffd640 10 monclient: dump: 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:51.984 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f48a7640 4 set_mon_vals no callback set 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 10 monclient: _renew_subs 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f01449c0 con 0x7f86f0142800 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.981+0000 7f86f830e640 1 librados: init done 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.982+0000 7f86e4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.982+0000 7f86e4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.982+0000 7f86e4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:51.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.982+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86cc005080 con 0x7f86f0142800 2024-03-29T11:35:51.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.984+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f86ec019460 con 0x7f86f0142800 2024-03-29T11:35:51.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.985+0000 7f86f60aa640 10 monclient: get_auth_request con 0x7f86d406e6a0 auth_method 0 2024-03-29T11:35:51.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.985+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f86ec053080 con 0x7f86f0142800 2024-03-29T11:35:51.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.985+0000 7f86e6ffd640 10 client.4365.objecter ms_dispatch 0x7f86f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:51.986 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.985+0000 7f86e6ffd640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:51.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.985+0000 7f86e6ffd640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:51.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.986+0000 7f86e6ffd640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-29T11:35:51.987 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.986+0000 7f86e6ffd640 10 client.4365.objecter ms_handle_connect 0x7f86d406e6a0 2024-03-29T11:35:51.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.993+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86ec04f320 con 0x7f86f0142800 2024-03-29T11:35:51.991 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.993+0000 7f86e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:51.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:51.993+0000 7f86e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:52.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.313+0000 7f86e4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:52.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.313+0000 7f86e4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:52.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.313+0000 7f86e4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:35:52.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.313+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86cc005330 con 0x7f86f0142800 2024-03-29T11:35:52.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.314+0000 7f86e6ffd640 1 -- 172.21.15.71:0/487948037 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f86ec04f320 con 0x7f86f0142800 2024-03-29T11:35:52.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.314+0000 7f86e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:52.313 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.314+0000 7f86e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:52.316 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:52.316 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.317+0000 7f86e4ff9640 10 librados: watch_flush enter 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 10 librados: watch_flush exit 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 10 client.4365.objecter successfully canceled tick 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 >> v1:172.21.15.71:6816/1596514235 conn(0x7f86d406e6a0 legacy=0x7f86d4070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 10 monclient: shutdown 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 >> v1:172.21.15.71:6789/0 conn(0x7f86f0142800 legacy=0x7f86f0071ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 shutdown_connections 2024-03-29T11:35:52.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 >> 172.21.15.71:0/487948037 conn(0x7f86f00b35e0 msgr2=0x7f86f0139050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:52.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 shutdown_connections 2024-03-29T11:35:52.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.318+0000 7f86e4ff9640 1 -- 172.21.15.71:0/487948037 wait complete. 2024-03-29T11:35:52.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:52.321+0000 7f86e4ff9640 1 librados: shutdown 2024-03-29T11:35:52.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:59.144 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.296+0000 7f81aa9bc640 10 monclient: get_monmap_and_config 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.296+0000 7f81aa9bc640 10 monclient: build_initial_monmap 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.296+0000 7f81aa9bc640 10 monclient: monmap: 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:35:59.297343+0000 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:35:59.297343+0000 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:35:59.296 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.297+0000 7f81aa9bc640 1 Processor -- start 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 1 -- start start 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient: init 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient(hunting): picked mon.noname-c con 0x7f81a40a9930 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient(hunting): picked mon.noname-a con 0x7f81a4149dc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient(hunting): picked mon.noname-b con 0x7f81a414ff40 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40aae60 con 0x7f81a4149dc0 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40b4310 con 0x7f81a40a9930 2024-03-29T11:35:59.299 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a40afdd0 con 0x7f81a414ff40 2024-03-29T11:35:59.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:59.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.298+0000 7f81aa9bc640 10 monclient(hunting): authenticate will time out at 1013.452951s 2024-03-29T11:35:59.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.302+0000 7f81a8f59640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f81a414ff40 0x7f81a4152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33959/0 (socket says 172.21.15.71:33959) 2024-03-29T11:35:59.300 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.302+0000 7f81a8f59640 1 -- 172.21.15.71:0/1005417627 learned_addr learned my addr 172.21.15.71:0/1005417627 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:59.302 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485446847 0 0) 0x7f81a40afdd0 con 0x7f81a414ff40 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c003520 con 0x7f81a414ff40 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197104881 0 0) 0x7f818c003520 con 0x7f81a414ff40 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 >> v1:172.21.15.81:6789/0 conn(0x7f81a40a9930 legacy=0x7f81a4143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.303+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 >> v1:172.21.15.71:6789/0 conn(0x7f81a4149dc0 legacy=0x7f81a414a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:59.303 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a414a950 con 0x7f81a414ff40 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81aa9bc640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81aa9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8194001690 con 0x7f81a414ff40 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: dump: 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.306 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81aa9bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 1 -- 172.21.15.71:0/1005417627 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8194001920 con 0x7f81a414ff40 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81aa9bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.304+0000 7f81aa9bc640 4 set_mon_vals no callback set 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.305+0000 7f81aa9bc640 10 monclient: shutdown 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.305+0000 7f81aa9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.305+0000 7f81aa9bc640 1 -- 172.21.15.71:0/1005417627 >> v1:172.21.15.71:6790/0 conn(0x7f81a414ff40 legacy=0x7f81a4152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.305+0000 7f81aa9bc640 1 -- 172.21.15.71:0/1005417627 shutdown_connections 2024-03-29T11:35:59.307 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.305+0000 7f81aa9bc640 1 -- 172.21.15.71:0/1005417627 >> 172.21.15.71:0/1005417627 conn(0x7f81a40b35e0 msgr2=0x7f81a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.306+0000 7f81aa9bc640 1 -- 172.21.15.71:0/1005417627 shutdown_connections 2024-03-29T11:35:59.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.307+0000 7f81aa9bc640 1 -- 172.21.15.71:0/1005417627 wait complete. 2024-03-29T11:35:59.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 10 monclient: build_initial_monmap 2024-03-29T11:35:59.308 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 10 monclient: monmap: 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 1 librados: starting msgr at 2024-03-29T11:35:59.309 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 1 librados: starting objecter 2024-03-29T11:35:59.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 1 Processor -- start 2024-03-29T11:35:59.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 1 -- start start 2024-03-29T11:35:59.311 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 1 librados: setting wanted keys 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 1 librados: calling monclient init 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.308+0000 7f81aa9bc640 10 monclient: init 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient(hunting): picked mon.noname-b con 0x7f81a40a9930 addr v1:172.21.15.81:6789/0 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient(hunting): picked mon.noname-c con 0x7f81a4149dc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient(hunting): picked mon.noname-a con 0x7f81a414ff40 addr v1:172.21.15.71:6789/0 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a413f3a0 con 0x7f81a414ff40 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a413f830 con 0x7f81a40a9930 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a406f3c0 con 0x7f81a4149dc0 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:59.312 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.309+0000 7f81aa9bc640 10 monclient(hunting): authenticate will time out at 1013.463606s 2024-03-29T11:35:59.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f81a40a9930 0x7f81a41e7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:53123/0 (socket says 172.21.15.71:53123) 2024-03-29T11:35:59.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a3fff640 1 -- 172.21.15.71:0/883531102 learned_addr learned my addr 172.21.15.71:0/883531102 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a0ff9640 10 client.?.objecter ms_handle_connect 0x7f81a40a9930 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a0ff9640 10 client.?.objecter ms_handle_connect 0x7f81a414ff40 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a0ff9640 10 client.?.objecter ms_handle_connect 0x7f81a4149dc0 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.315+0000 7f81a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850266350 0 0) 0x7f81a413f3a0 con 0x7f81a414ff40 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8190004eb0 con 0x7f81a414ff40 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904597703 0 0) 0x7f8190004eb0 con 0x7f81a414ff40 2024-03-29T11:35:59.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a413f3a0 con 0x7f81a414ff40 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8194004980 con 0x7f81a414ff40 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:59.318 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.316+0000 7f81a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715115653 0 0) 0x7f81a406f3c0 con 0x7f81a4149dc0 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8190004eb0 con 0x7f81a4149dc0 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257192498 0 0) 0x7f81a413f830 con 0x7f81a40a9930 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.317+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a406f3c0 con 0x7f81a40a9930 2024-03-29T11:35:59.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1289612140 0 0) 0x7f8190004eb0 con 0x7f81a4149dc0 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a413f830 con 0x7f81a4149dc0 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f819c001b50 con 0x7f81a4149dc0 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:59.320 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549377626 0 0) 0x7f81a406f3c0 con 0x7f81a40a9930 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8190004eb0 con 0x7f81a40a9930 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f81980018a0 con 0x7f81a40a9930 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:59.323 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3569115917 0 0) 0x7f81a413f830 con 0x7f81a4149dc0 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 >> v1:172.21.15.81:6789/0 conn(0x7f81a40a9930 legacy=0x7f81a41e7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 >> v1:172.21.15.71:6789/0 conn(0x7f81a414ff40 legacy=0x7f81a4071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 monclient: found mon.c 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:59.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4140060 con 0x7f81a4149dc0 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.318+0000 7f81a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f819c002040 con 0x7f81a4149dc0 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f819c003b10 con 0x7f81a4149dc0 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81a0ff9640 10 monclient: dump: 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:35:59.325 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 10 monclient: _renew_subs 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a411f200 con 0x7f81a4149dc0 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 10 monclient: _renew_subs 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a413fa40 con 0x7f81a4149dc0 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.319+0000 7f81aa9bc640 1 librados: init done 2024-03-29T11:35:59.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.320+0000 7f81a27fc640 4 set_mon_vals no callback set 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.322+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f819c019ae0 con 0x7f81a4149dc0 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.322+0000 7f81aa9bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.322+0000 7f81aa9bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.322+0000 7f81aa9bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.322+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8184005080 con 0x7f81a4149dc0 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a3fff640 10 monclient: get_auth_request con 0x7f819006e110 auth_method 0 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f819c083ac0 con 0x7f81a4149dc0 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a0ff9640 10 client.4391.objecter ms_dispatch 0x7f81a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a0ff9640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a0ff9640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a0ff9640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T11:35:59.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.323+0000 7f81a0ff9640 10 client.4391.objecter ms_handle_connect 0x7f819006e110 2024-03-29T11:35:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.331+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f819c085e60 con 0x7f81a4149dc0 2024-03-29T11:35:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.331+0000 7f81a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:59.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.331+0000 7f81a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:59.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.661+0000 7f81aa9bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:59.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.661+0000 7f81aa9bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:59.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.661+0000 7f81aa9bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:35:59.660 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.661+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8184005330 con 0x7f81a4149dc0 2024-03-29T11:35:59.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.663+0000 7f81a0ff9640 1 -- 172.21.15.71:0/883531102 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f819c04f1d0 con 0x7f81a4149dc0 2024-03-29T11:35:59.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.663+0000 7f81a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:59.661 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.663+0000 7f81a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:59.663 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:35:59.663 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:59.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.668+0000 7f81aa9bc640 10 librados: watch_flush enter 2024-03-29T11:35:59.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.668+0000 7f81aa9bc640 10 librados: watch_flush exit 2024-03-29T11:35:59.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.668+0000 7f81aa9bc640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T11:35:59.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.668+0000 7f81aa9bc640 10 client.4391.objecter successfully canceled tick 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 >> v1:172.21.15.71:6816/1596514235 conn(0x7f819006e110 legacy=0x7f81900705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 10 monclient: shutdown 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 >> v1:172.21.15.71:6790/0 conn(0x7f81a4149dc0 legacy=0x7f81a40ac5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 shutdown_connections 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 >> 172.21.15.71:0/883531102 conn(0x7f81a40b35e0 msgr2=0x7f81a40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 shutdown_connections 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 -- 172.21.15.71:0/883531102 wait complete. 2024-03-29T11:35:59.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:35:59.669+0000 7f81aa9bc640 1 librados: shutdown 2024-03-29T11:35:59.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:06.492 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:06.646 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.647+0000 7fc73cc8f640 10 monclient: get_monmap_and_config 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.647+0000 7fc73cc8f640 10 monclient: build_initial_monmap 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.647+0000 7fc73cc8f640 10 monclient: monmap: 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:06.648315+0000 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:06.648315+0000 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:06.647 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:06.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.649+0000 7fc73cc8f640 1 Processor -- start 2024-03-29T11:36:06.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.649+0000 7fc73cc8f640 1 -- start start 2024-03-29T11:36:06.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.649+0000 7fc73cc8f640 10 monclient: init 2024-03-29T11:36:06.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:06.649 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7380a9930 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc738149dc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc73814ff40 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7380aae60 con 0x7fc73814ff40 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7380b4310 con 0x7fc7380a9930 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7380afdd0 con 0x7fc738149dc0 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:06.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.650+0000 7fc73cc8f640 10 monclient(hunting): authenticate will time out at 1020.804589s 2024-03-29T11:36:06.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.654+0000 7fc735d9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fc738149dc0 0x7fc73814a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36403/0 (socket says 172.21.15.71:36403) 2024-03-29T11:36:06.653 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.654+0000 7fc735d9b640 1 -- 172.21.15.71:0/2025087903 learned_addr learned my addr 172.21.15.71:0/2025087903 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:06.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.657+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188559755 0 0) 0x7fc7380afdd0 con 0x7fc738149dc0 2024-03-29T11:36:06.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.657+0000 7fc73559a640 10 monclient(hunting): _negotiate 2024-03-29T11:36:06.656 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.657+0000 7fc73559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:06.657 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.657+0000 7fc73559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:06.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.657+0000 7fc73559a640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.657+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc720003520 con 0x7fc738149dc0 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600060590 0 0) 0x7fc720003520 con 0x7fc738149dc0 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 >> v1:172.21.15.81:6789/0 conn(0x7fc7380a9930 legacy=0x7fc738143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 >> v1:172.21.15.71:6789/0 conn(0x7fc73814ff40 legacy=0x7fc738152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: _finish_hunting 0 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 monclient: found mon.noname-b 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73814a950 con 0x7fc738149dc0 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: _finish_auth 0 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: _check_auth_tickets 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73cc8f640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:36:06.665 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73cc8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc728001690 con 0x7fc738149dc0 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: dump: 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:06.666 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73cc8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc728001870 con 0x7fc738149dc0 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73cc8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73cc8f640 4 set_mon_vals no callback set 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 1 -- 172.21.15.71:0/2025087903 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc728001cc0 con 0x7fc738149dc0 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.658+0000 7fc73559a640 10 monclient: dump: 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.667 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.659+0000 7fc73cc8f640 10 monclient: shutdown 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.659+0000 7fc73cc8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.659+0000 7fc73cc8f640 1 -- 172.21.15.71:0/2025087903 >> v1:172.21.15.71:6790/0 conn(0x7fc738149dc0 legacy=0x7fc73814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.659+0000 7fc73cc8f640 1 -- 172.21.15.71:0/2025087903 shutdown_connections 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.659+0000 7fc73cc8f640 1 -- 172.21.15.71:0/2025087903 >> 172.21.15.71:0/2025087903 conn(0x7fc7380b35e0 msgr2=0x7fc7380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.661+0000 7fc73cc8f640 1 -- 172.21.15.71:0/2025087903 shutdown_connections 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.661+0000 7fc73cc8f640 1 -- 172.21.15.71:0/2025087903 wait complete. 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.665+0000 7fc73cc8f640 10 monclient: build_initial_monmap 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.665+0000 7fc73cc8f640 10 monclient: monmap: 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:06.668 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.665+0000 7fc73cc8f640 1 librados: starting msgr at 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.665+0000 7fc73cc8f640 1 librados: starting objecter 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.665+0000 7fc73cc8f640 1 Processor -- start 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 1 -- start start 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 1 librados: setting wanted keys 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 1 librados: calling monclient init 2024-03-29T11:36:06.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient: init 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7380a9930 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc738149dc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc73814ff40 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7380aca00 con 0x7fc738149dc0 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7380ac520 con 0x7fc7380a9930 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73806e840 con 0x7fc73814ff40 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:06.672 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.666+0000 7fc73cc8f640 10 monclient(hunting): authenticate will time out at 1020.820938s 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc736d9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fc73814ff40 0x7fc7382000a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36487/0 (socket says 172.21.15.71:36487) 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc736d9d640 1 -- 172.21.15.71:0/1806258566 learned_addr learned my addr 172.21.15.71:0/1806258566 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 10 client.?.objecter ms_handle_connect 0x7fc73814ff40 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957136201 0 0) 0x7fc73806e840 con 0x7fc73814ff40 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.671+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc718004eb0 con 0x7fc73814ff40 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251694580 0 0) 0x7fc718004eb0 con 0x7fc73814ff40 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc73806e840 con 0x7fc73814ff40 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc730001b50 con 0x7fc73814ff40 2024-03-29T11:36:06.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 10 monclient(hunting): dump: 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.672+0000 7fc726ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:06.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.673+0000 7fc726ffd640 10 client.?.objecter ms_handle_connect 0x7fc7380a9930 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.673+0000 7fc726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.673+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60531859 0 0) 0x7fc73806e840 con 0x7fc73814ff40 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.673+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 >> v1:172.21.15.81:6789/0 conn(0x7fc7380a9930 legacy=0x7fc738137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.673+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 >> v1:172.21.15.71:6789/0 conn(0x7fc738149dc0 legacy=0x7fc73813c3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 1 monclient: found mon.c 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:06.675 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73813b120 con 0x7fc73814ff40 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 10 monclient: _renew_subs 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc738137b30 con 0x7fc73814ff40 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 10 monclient: _renew_subs 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73813bb80 con 0x7fc73814ff40 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.674+0000 7fc73cc8f640 1 librados: init done 2024-03-29T11:36:06.676 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.675+0000 7fc724ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:06.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.675+0000 7fc724ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:06.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.675+0000 7fc724ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:06.677 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.675+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc738075f10 con 0x7fc73814ff40 2024-03-29T11:36:06.678 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc730001cb0 con 0x7fc73814ff40 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fc730003b30 con 0x7fc73814ff40 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 10 monclient: dump: 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:06.679 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.677+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fc730003d50 con 0x7fc73814ff40 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc734d99640 4 set_mon_vals no callback set 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fc730084dd0 con 0x7fc73814ff40 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc726ffd640 10 client.4397.objecter ms_dispatch 0x7fc738000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc726ffd640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc726ffd640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc726ffd640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.678+0000 7fc73659c640 10 monclient: get_auth_request con 0x7fc71806dd10 auth_method 0 2024-03-29T11:36:06.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.679+0000 7fc726ffd640 10 client.4397.objecter ms_handle_connect 0x7fc71806dd10 2024-03-29T11:36:06.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.684+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc730050730 con 0x7fc73814ff40 2024-03-29T11:36:06.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.684+0000 7fc726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:06.682 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:06.684+0000 7fc726ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:07.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.008+0000 7fc724ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:07.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.008+0000 7fc724ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:07.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.008+0000 7fc724ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:07.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.008+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc73813b2b0 con 0x7fc73814ff40 2024-03-29T11:36:07.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.009+0000 7fc726ffd640 1 -- 172.21.15.71:0/1806258566 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fc730050730 con 0x7fc73814ff40 2024-03-29T11:36:07.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.009+0000 7fc726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:07.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.009+0000 7fc726ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:07.014 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:07.014 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:07.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 10 librados: watch_flush enter 2024-03-29T11:36:07.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 10 librados: watch_flush exit 2024-03-29T11:36:07.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 10 client.4397.objecter successfully canceled tick 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 >> v1:172.21.15.71:6816/1596514235 conn(0x7fc71806dd10 legacy=0x7fc7180701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 10 monclient: shutdown 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.016+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 >> v1:172.21.15.71:6790/0 conn(0x7fc73814ff40 legacy=0x7fc7382000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.017+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 shutdown_connections 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.017+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 >> 172.21.15.71:0/1806258566 conn(0x7fc7380b35e0 msgr2=0x7fc738140ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.017+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 shutdown_connections 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.017+0000 7fc724ff9640 1 -- 172.21.15.71:0/1806258566 wait complete. 2024-03-29T11:36:07.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:07.017+0000 7fc724ff9640 1 librados: shutdown 2024-03-29T11:36:07.034 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:13.841 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:13.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.994+0000 7f4a31b54640 10 monclient: get_monmap_and_config 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.994+0000 7f4a31b54640 10 monclient: build_initial_monmap 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.994+0000 7f4a31b54640 10 monclient: monmap: 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:13.995420+0000 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:13.995420+0000 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:13.994 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:13.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.995+0000 7f4a31b54640 1 Processor -- start 2024-03-29T11:36:13.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.996+0000 7f4a31b54640 1 -- start start 2024-03-29T11:36:13.997 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.996+0000 7f4a31b54640 10 monclient: init 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient(hunting): picked mon.noname-a con 0x7f4a2c0a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient(hunting): picked mon.noname-b con 0x7f4a2c0aacf0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient(hunting): picked mon.noname-c con 0x7f4a2c0af220 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0ac2f0 con 0x7f4a2c0a9750 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0acaf0 con 0x7f4a2c0af220 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0ac710 con 0x7f4a2c0aacf0 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.997+0000 7f4a31b54640 10 monclient(hunting): authenticate will time out at 1028.151779s 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4a2c0a9750 0x7f4a2c0ad010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49686/0 (socket says 172.21.15.71:49686) 2024-03-29T11:36:13.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2b7fe640 1 -- 172.21.15.71:0/354960023 learned_addr learned my addr 172.21.15.71:0/354960023 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765365860 0 0) 0x7f4a2c0ac2f0 con 0x7f4a2c0a9750 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2a7fc640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:13.999+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a14003520 con 0x7f4a2c0a9750 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.000+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726908014 0 0) 0x7f4a14003520 con 0x7f4a2c0a9750 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.000+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 >> v1:172.21.15.71:6790/0 conn(0x7f4a2c0aacf0 legacy=0x7f4a2c13e6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:13.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.000+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 >> v1:172.21.15.81:6789/0 conn(0x7f4a2c0af220 legacy=0x7f4a2c149f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c0ac8a0 con 0x7f4a2c0a9750 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4a20001690 con 0x7f4a2c0a9750 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a2a7fc640 10 monclient: dump: 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a31b54640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:36:14.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.002+0000 7f4a31b54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:14.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a20001860 con 0x7f4a2c0a9750 2024-03-29T11:36:14.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:14.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a2a7fc640 1 -- 172.21.15.71:0/354960023 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4a20001cb0 con 0x7f4a2c0a9750 2024-03-29T11:36:14.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:14.005 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a2a7fc640 10 monclient: dump: 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a31b54640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.003+0000 7f4a31b54640 4 set_mon_vals no callback set 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.004+0000 7f4a31b54640 10 monclient: shutdown 2024-03-29T11:36:14.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.004+0000 7f4a31b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:14.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.004+0000 7f4a31b54640 1 -- 172.21.15.71:0/354960023 >> v1:172.21.15.71:6789/0 conn(0x7f4a2c0a9750 legacy=0x7f4a2c0ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.005+0000 7f4a31b54640 1 -- 172.21.15.71:0/354960023 shutdown_connections 2024-03-29T11:36:14.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.005+0000 7f4a31b54640 1 -- 172.21.15.71:0/354960023 >> 172.21.15.71:0/354960023 conn(0x7f4a2c06b660 msgr2=0x7f4a2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.006+0000 7f4a31b54640 1 -- 172.21.15.71:0/354960023 shutdown_connections 2024-03-29T11:36:14.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.006+0000 7f4a31b54640 1 -- 172.21.15.71:0/354960023 wait complete. 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.007+0000 7f4a31b54640 10 monclient: build_initial_monmap 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.007+0000 7f4a31b54640 10 monclient: monmap: 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:14.008 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.007+0000 7f4a31b54640 1 librados: starting msgr at 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.007+0000 7f4a31b54640 1 librados: starting objecter 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.008+0000 7f4a31b54640 1 Processor -- start 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.008+0000 7f4a31b54640 1 -- start start 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.008+0000 7f4a31b54640 1 librados: setting wanted keys 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.008+0000 7f4a31b54640 1 librados: calling monclient init 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.008+0000 7f4a31b54640 10 monclient: init 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:14.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient(hunting): picked mon.noname-b con 0x7f4a2c0a9750 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient(hunting): picked mon.noname-c con 0x7f4a2c0aacf0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient(hunting): picked mon.noname-a con 0x7f4a2c0af220 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c136180 con 0x7f4a2c0af220 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c071a50 con 0x7f4a2c0a9750 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c06f9b0 con 0x7f4a2c0aacf0 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a31b54640 10 monclient(hunting): authenticate will time out at 1028.163634s 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a2bfff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f4a2c0af220 0x7f4a2c135a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49986/0 (socket says 172.21.15.71:49986) 2024-03-29T11:36:14.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.009+0000 7f4a2bfff640 1 -- 172.21.15.71:0/390055755 learned_addr learned my addr 172.21.15.71:0/390055755 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 10 client.?.objecter ms_handle_connect 0x7f4a2c0af220 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584191577 0 0) 0x7f4a2c136180 con 0x7f4a2c0af220 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:14.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.010+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a18004eb0 con 0x7f4a2c0af220 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4255846744 0 0) 0x7f4a18004eb0 con 0x7f4a2c0af220 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a2c136180 con 0x7f4a2c0af220 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4a24001ad0 con 0x7f4a2c0af220 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 10 monclient(hunting): dump: 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:14.013 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:14.014 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:14.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:14.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2737740747 0 0) 0x7f4a2c136180 con 0x7f4a2c0af220 2024-03-29T11:36:14.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.011+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 >> v1:172.21.15.71:6790/0 conn(0x7f4a2c0aacf0 legacy=0x7f4a2c135360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:14.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.013+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 >> v1:172.21.15.81:6789/0 conn(0x7f4a2c0a9750 legacy=0x7f4a2c06d9d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 1 monclient: found mon.a 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c070690 con 0x7f4a2c0af220 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a31b54640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:36:14.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a31b54640 10 monclient: _renew_subs 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a31b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.014+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c06e0e0 con 0x7f4a2c0af220 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.015+0000 7f4a31b54640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.015+0000 7f4a31b54640 10 monclient: _renew_subs 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.015+0000 7f4a31b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.015+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c070d90 con 0x7f4a2c0af220 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.015+0000 7f4a31b54640 1 librados: init done 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.016+0000 7f4a31b54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.016+0000 7f4a31b54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.016+0000 7f4a31b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:14.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.016+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c1c4c10 con 0x7f4a2c0af220 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a24001c30 con 0x7f4a2c0af220 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4a24004d80 con 0x7f4a2c0af220 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a13fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a13fff640 10 monclient: dump: 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.018 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.017+0000 7f4a29ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f4a2401add0 con 0x7f4a2c0af220 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a2b7fe640 10 monclient: get_auth_request con 0x7f4a1806dcd0 auth_method 0 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f4a24084d50 con 0x7f4a2c0af220 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a13fff640 10 client.4377.objecter ms_dispatch 0x7f4a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a13fff640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a13fff640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.018+0000 7f4a13fff640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:36:14.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.019+0000 7f4a13fff640 10 client.4377.objecter ms_handle_connect 0x7f4a1806dcd0 2024-03-29T11:36:14.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.026+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a240504a0 con 0x7f4a2c0af220 2024-03-29T11:36:14.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.026+0000 7f4a13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:14.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.026+0000 7f4a13fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:14.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.351+0000 7f4a31b54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:14.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.351+0000 7f4a31b54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:14.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.351+0000 7f4a31b54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:14.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.351+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a2c070820 con 0x7f4a2c0af220 2024-03-29T11:36:14.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.352+0000 7f4a13fff640 1 -- 172.21.15.71:0/390055755 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f4a24052560 con 0x7f4a2c0af220 2024-03-29T11:36:14.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.352+0000 7f4a13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:14.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.352+0000 7f4a13fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:14.353 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:14.353 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:14.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 10 librados: watch_flush enter 2024-03-29T11:36:14.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 10 librados: watch_flush exit 2024-03-29T11:36:14.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:36:14.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:36:14.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 >> v1:172.21.15.71:6816/1596514235 conn(0x7f4a1806dcd0 legacy=0x7f4a18070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 10 monclient: shutdown 2024-03-29T11:36:14.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:14.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 >> v1:172.21.15.71:6789/0 conn(0x7f4a2c0af220 legacy=0x7f4a2c135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 shutdown_connections 2024-03-29T11:36:14.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.356+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 >> 172.21.15.71:0/390055755 conn(0x7f4a2c06b660 msgr2=0x7f4a2c0b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.357+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 shutdown_connections 2024-03-29T11:36:14.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.357+0000 7f4a31b54640 1 -- 172.21.15.71:0/390055755 wait complete. 2024-03-29T11:36:14.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:14.357+0000 7f4a31b54640 1 librados: shutdown 2024-03-29T11:36:14.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:21.180 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:21.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.330+0000 7fe7d4b6d640 10 monclient: get_monmap_and_config 2024-03-29T11:36:21.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.330+0000 7fe7d4b6d640 10 monclient: build_initial_monmap 2024-03-29T11:36:21.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.330+0000 7fe7d4b6d640 10 monclient: monmap: 2024-03-29T11:36:21.329 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:21.329 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:21.329 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:21.331005+0000 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:21.331005+0000 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:21.330 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.331+0000 7fe7d4b6d640 1 Processor -- start 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.331+0000 7fe7d4b6d640 1 -- start start 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.331+0000 7fe7d4b6d640 10 monclient: init 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c80389e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:21.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c8038dc0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c8030660 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c80c54c0 con 0x7fe7c80389e0 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c8036430 con 0x7fe7c8030660 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c8039840 con 0x7fe7c8038dc0 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7d4b6d640 10 monclient(hunting): authenticate will time out at 1035.486674s 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7ceffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe7c8038dc0 0x7fe7c80cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:44918/0 (socket says 172.21.15.71:44918) 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.332+0000 7fe7ceffd640 1 -- 172.21.15.71:0/1142956381 learned_addr learned my addr 172.21.15.71:0/1142956381 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.333+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699181570 0 0) 0x7fe7c8036430 con 0x7fe7c8030660 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.333+0000 7fe7ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.333+0000 7fe7ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.333+0000 7fe7ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.333+0000 7fe7ce7fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:36:21.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.333+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c0003520 con 0x7fe7c8030660 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 751228792 0 0) 0x7fe7c0003520 con 0x7fe7c8030660 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 >> v1:172.21.15.71:6790/0 conn(0x7fe7c8038dc0 legacy=0x7fe7c80cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 >> v1:172.21.15.71:6789/0 conn(0x7fe7c80389e0 legacy=0x7fe7c8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 monclient: found mon.noname-c 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c8035690 con 0x7fe7c8030660 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:21.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7d4b6d640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7d4b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7d004f730 con 0x7fe7c8030660 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: dump: 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.337 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7d4b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7d00008d0 con 0x7fe7c8030660 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7d4b6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7d4b6d640 4 set_mon_vals no callback set 2024-03-29T11:36:21.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 1 -- 172.21.15.71:0/1142956381 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7d004f730 con 0x7fe7c8030660 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.334+0000 7fe7ce7fc640 10 monclient: dump: 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:21.339 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.335+0000 7fe7d4b6d640 10 monclient: shutdown 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.335+0000 7fe7d4b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.335+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/1142956381 >> v1:172.21.15.81:6789/0 conn(0x7fe7c8030660 legacy=0x7fe7c80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.335+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/1142956381 shutdown_connections 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.335+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/1142956381 >> 172.21.15.71:0/1142956381 conn(0x7fe7c8005c00 msgr2=0x7fe7c8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.335+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/1142956381 shutdown_connections 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.336+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/1142956381 wait complete. 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.337+0000 7fe7d4b6d640 10 monclient: build_initial_monmap 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.337+0000 7fe7d4b6d640 10 monclient: monmap: 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:21.340 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:21.341 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.337+0000 7fe7d4b6d640 1 librados: starting msgr at 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.337+0000 7fe7d4b6d640 1 librados: starting objecter 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.337+0000 7fe7d4b6d640 1 Processor -- start 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.338+0000 7fe7d4b6d640 1 -- start start 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.338+0000 7fe7d4b6d640 1 librados: setting wanted keys 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.338+0000 7fe7d4b6d640 1 librados: calling monclient init 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.338+0000 7fe7d4b6d640 10 monclient: init 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:21.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe7c8030660 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe7c80389e0 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe7c802ebd0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c80c03e0 con 0x7fe7c802ebd0 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c80c8580 con 0x7fe7c80389e0 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7c80c30e0 con 0x7fe7c8030660 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7d4b6d640 10 monclient(hunting): authenticate will time out at 1035.493686s 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7ceffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe7c80389e0 0x7fe7c80bf5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:52842/0 (socket says 172.21.15.71:52842) 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.339+0000 7fe7ceffd640 1 -- 172.21.15.71:0/4218347093 learned_addr learned my addr 172.21.15.71:0/4218347093 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 client.?.objecter ms_handle_connect 0x7fe7c80389e0 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:21.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734198096 0 0) 0x7fe7c80c8580 con 0x7fe7c80389e0 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b4004eb0 con 0x7fe7c80389e0 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 client.?.objecter ms_handle_connect 0x7fe7c8030660 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 client.?.objecter ms_handle_connect 0x7fe7c802ebd0 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277241476 0 0) 0x7fe7c80c30e0 con 0x7fe7c8030660 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.340+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c80c8580 con 0x7fe7c8030660 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3992910636 0 0) 0x7fe7b4004eb0 con 0x7fe7c80389e0 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c80c30e0 con 0x7fe7c80389e0 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7c4001890 con 0x7fe7c80389e0 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): dump: 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:21.352 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877247264 0 0) 0x7fe7c80c03e0 con 0x7fe7c802ebd0 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b4004eb0 con 0x7fe7c802ebd0 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 399984484 0 0) 0x7fe7c80c8580 con 0x7fe7c8030660 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7c80c03e0 con 0x7fe7c8030660 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.341+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7bc001820 con 0x7fe7c8030660 2024-03-29T11:36:21.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient(hunting): dump: 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1227463954 0 0) 0x7fe7c80c30e0 con 0x7fe7c80389e0 2024-03-29T11:36:21.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 >> v1:172.21.15.71:6790/0 conn(0x7fe7c8030660 legacy=0x7fe7c81618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 >> v1:172.21.15.71:6789/0 conn(0x7fe7c802ebd0 legacy=0x7fe7c80bfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 1 monclient: found mon.b 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c8162000 con 0x7fe7c80389e0 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 10 monclient: _renew_subs 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c80c3b30 con 0x7fe7c80389e0 2024-03-29T11:36:21.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 10 monclient: _renew_subs 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.342+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c802d150 con 0x7fe7c80389e0 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7d4b6d640 1 librados: init done 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7c40047e0 con 0x7fe7c80389e0 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7c4001d80 con 0x7fe7c80389e0 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7bbfff640 10 monclient: dump: 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.356 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.343+0000 7fe7cdffb640 4 set_mon_vals no callback set 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.344+0000 7fe7d4b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.344+0000 7fe7d4b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.344+0000 7fe7d4b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.344+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b0005080 con 0x7fe7c80389e0 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.346+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fe7c401aa60 con 0x7fe7c80389e0 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fe7c40849a0 con 0x7fe7c80389e0 2024-03-29T11:36:21.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7cf7fe640 10 monclient: get_auth_request con 0x7fe7b406dff0 auth_method 0 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7bbfff640 10 client.4390.objecter ms_dispatch 0x7fe7d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7bbfff640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7bbfff640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7bbfff640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.348+0000 7fe7bbfff640 10 client.4390.objecter ms_handle_connect 0x7fe7b406dff0 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.353+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7c4050280 con 0x7fe7c80389e0 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.353+0000 7fe7bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:21.358 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.353+0000 7fe7bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:21.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.732+0000 7fe7d4b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:21.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.732+0000 7fe7d4b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:21.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.732+0000 7fe7d4b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:36:21.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.732+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7b0005330 con 0x7fe7c80389e0 2024-03-29T11:36:21.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.733+0000 7fe7bbfff640 1 -- 172.21.15.71:0/4218347093 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fe7c40521b0 con 0x7fe7c80389e0 2024-03-29T11:36:21.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.733+0000 7fe7bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:21.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.733+0000 7fe7bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:21.733 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:21.733 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:21.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 10 librados: watch_flush enter 2024-03-29T11:36:21.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 10 librados: watch_flush exit 2024-03-29T11:36:21.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-29T11:36:21.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 10 client.4390.objecter successfully canceled tick 2024-03-29T11:36:21.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 >> v1:172.21.15.71:6816/1596514235 conn(0x7fe7b406dff0 legacy=0x7fe7b40704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 10 monclient: shutdown 2024-03-29T11:36:21.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:21.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 >> v1:172.21.15.81:6789/0 conn(0x7fe7c80389e0 legacy=0x7fe7c80bf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 shutdown_connections 2024-03-29T11:36:21.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 >> 172.21.15.71:0/4218347093 conn(0x7fe7c8005c00 msgr2=0x7fe7c803ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:21.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.737+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 shutdown_connections 2024-03-29T11:36:21.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.738+0000 7fe7d4b6d640 1 -- 172.21.15.71:0/4218347093 wait complete. 2024-03-29T11:36:21.737 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:21.738+0000 7fe7d4b6d640 1 librados: shutdown 2024-03-29T11:36:21.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:28.560 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:28.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.694+0000 7f7f57139640 10 monclient: get_monmap_and_config 2024-03-29T11:36:28.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.694+0000 7f7f57139640 10 monclient: build_initial_monmap 2024-03-29T11:36:28.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.694+0000 7f7f57139640 10 monclient: monmap: 2024-03-29T11:36:28.693 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:28.693 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:28.695132+0000 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:28.695132+0000 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:28.694 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:28.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.695+0000 7f7f57139640 1 Processor -- start 2024-03-29T11:36:28.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.695+0000 7f7f57139640 1 -- start start 2024-03-29T11:36:28.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.695+0000 7f7f57139640 10 monclient: init 2024-03-29T11:36:28.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.696+0000 7f7f57139640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:28.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.696+0000 7f7f57139640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:28.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.696+0000 7f7f57139640 10 monclient(hunting): picked mon.noname-c con 0x7f7f501457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:28.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.696+0000 7f7f57139640 10 monclient(hunting): picked mon.noname-a con 0x7f7f5013f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:28.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.696+0000 7f7f54ed5640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f7f501457f0 0x7f7f5013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:49336/0 (socket says 172.21.15.71:49336) 2024-03-29T11:36:28.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f57139640 10 monclient(hunting): picked mon.noname-b con 0x7f7f500a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:28.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f57139640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f500aaef0 con 0x7f7f5013f650 2024-03-29T11:36:28.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f57139640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f50145bd0 con 0x7f7f501457f0 2024-03-29T11:36:28.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f556d6640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7f500a9d70 0x7f7f5014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33658/0 (socket says 172.21.15.71:33658) 2024-03-29T11:36:28.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f556d6640 1 -- 172.21.15.71:0/1088797429 learned_addr learned my addr 172.21.15.71:0/1088797429 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:28.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f57139640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f50145eb0 con 0x7f7f500a9d70 2024-03-29T11:36:28.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f57139640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:28.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.698+0000 7f7f57139640 10 monclient(hunting): authenticate will time out at 1042.852480s 2024-03-29T11:36:28.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064496609 0 0) 0x7f7f50145eb0 con 0x7f7f500a9d70 2024-03-29T11:36:28.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:36:28.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f40003520 con 0x7f7f500a9d70 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650540712 0 0) 0x7f7f50145bd0 con 0x7f7f501457f0 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.699+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f50145eb0 con 0x7f7f501457f0 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081054414 0 0) 0x7f7f40003520 con 0x7f7f500a9d70 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 >> v1:172.21.15.81:6789/0 conn(0x7f7f501457f0 legacy=0x7f7f5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 >> v1:172.21.15.71:6789/0 conn(0x7f7f5013f650 legacy=0x7f7f500b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f50146340 con 0x7f7f500a9d70 2024-03-29T11:36:28.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7f48001a10 con 0x7f7f500a9d70 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: dump: 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:28.701 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f57139640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f57139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f48001b70 con 0x7f7f500a9d70 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f57139640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f57139640 4 set_mon_vals no callback set 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 1 -- 172.21.15.71:0/1088797429 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7f48001fc0 con 0x7f7f500a9d70 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.700+0000 7f7f4f7fe640 10 monclient: dump: 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:28.702 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:28.703 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 10 monclient: shutdown 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 1 -- 172.21.15.71:0/1088797429 >> v1:172.21.15.71:6790/0 conn(0x7f7f500a9d70 legacy=0x7f7f5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 1 -- 172.21.15.71:0/1088797429 shutdown_connections 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 1 -- 172.21.15.71:0/1088797429 >> 172.21.15.71:0/1088797429 conn(0x7f7f50066a00 msgr2=0x7f7f50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 1 -- 172.21.15.71:0/1088797429 shutdown_connections 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.701+0000 7f7f57139640 1 -- 172.21.15.71:0/1088797429 wait complete. 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.702+0000 7f7f57139640 10 monclient: build_initial_monmap 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.702+0000 7f7f57139640 10 monclient: monmap: 2024-03-29T11:36:28.709 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 1 librados: starting msgr at 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 1 librados: starting objecter 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 1 Processor -- start 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 1 -- start start 2024-03-29T11:36:28.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 1 librados: setting wanted keys 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 1 librados: calling monclient init 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.703+0000 7f7f57139640 10 monclient: init 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.704+0000 7f7f57139640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.704+0000 7f7f57139640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.704+0000 7f7f57139640 10 monclient(hunting): picked mon.noname-c con 0x7f7f500a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.704+0000 7f7f57139640 10 monclient(hunting): picked mon.noname-b con 0x7f7f5013f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.704+0000 7f7f54ed5640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7f500a9d70 0x7f7f5006e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33928/0 (socket says 172.21.15.71:33928) 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.704+0000 7f7f54ed5640 1 -- 172.21.15.71:0/3459675369 learned_addr learned my addr 172.21.15.71:0/3459675369 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.705+0000 7f7f57139640 10 monclient(hunting): picked mon.noname-a con 0x7f7f501457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.705+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f501e7860 con 0x7f7f501457f0 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.705+0000 7f7f4d7fa640 10 client.?.objecter ms_handle_connect 0x7f7f500a9d70 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.705+0000 7f7f4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:28.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.705+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5006e810 con 0x7f7f5013f650 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.705+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5006f800 con 0x7f7f500a9d70 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 client.?.objecter ms_handle_connect 0x7f7f5013f650 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983976543 0 0) 0x7f7f5006f800 con 0x7f7f500a9d70 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f57139640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f57139640 10 monclient(hunting): authenticate will time out at 1042.860240s 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f34004eb0 con 0x7f7f500a9d70 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 273511804 0 0) 0x7f7f34004eb0 con 0x7f7f500a9d70 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f5006f800 con 0x7f7f500a9d70 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7f44001910 con 0x7f7f500a9d70 2024-03-29T11:36:28.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): dump: 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.706+0000 7f7f4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:28.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753292808 0 0) 0x7f7f5006e810 con 0x7f7f5013f650 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f34004eb0 con 0x7f7f5013f650 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4257792804 0 0) 0x7f7f5006f800 con 0x7f7f500a9d70 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 >> v1:172.21.15.81:6789/0 conn(0x7f7f5013f650 legacy=0x7f7f501e7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.707+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 >> v1:172.21.15.71:6789/0 conn(0x7f7f501457f0 legacy=0x7f7f501eee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 1 monclient: found mon.c 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:28.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5013fda0 con 0x7f7f500a9d70 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 10 monclient: _renew_subs 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f50142350 con 0x7f7f500a9d70 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 10 monclient: _renew_subs 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f501e8400 con 0x7f7f500a9d70 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f440047e0 con 0x7f7f500a9d70 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f57139640 1 librados: init done 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:28.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7f44004af0 con 0x7f7f500a9d70 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4effd640 4 set_mon_vals no callback set 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.708+0000 7f7f4d7fa640 10 monclient: dump: 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:28.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.710+0000 7f7f57139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:28.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.710+0000 7f7f57139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:28.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.710+0000 7f7f57139640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:28.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.710+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f500b24a0 con 0x7f7f500a9d70 2024-03-29T11:36:28.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.710+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f7f4401ab40 con 0x7f7f500a9d70 2024-03-29T11:36:28.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f7f44084ba0 con 0x7f7f500a9d70 2024-03-29T11:36:28.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4d7fa640 10 client.4409.objecter ms_dispatch 0x7f7f50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:28.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4d7fa640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:28.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4d7fa640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:28.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4d7fa640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:36:28.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4ffff640 10 monclient: get_auth_request con 0x7f7f3406dd90 auth_method 0 2024-03-29T11:36:28.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.711+0000 7f7f4d7fa640 10 client.4409.objecter ms_handle_connect 0x7f7f3406dd90 2024-03-29T11:36:28.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.719+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f44050480 con 0x7f7f500a9d70 2024-03-29T11:36:28.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.719+0000 7f7f4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:28.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:28.719+0000 7f7f4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:29.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f57139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:29.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f57139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:29.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f57139640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:29.023 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f5006f600 con 0x7f7f500a9d70 2024-03-29T11:36:29.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f4d7fa640 1 -- 172.21.15.71:0/3459675369 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f7f440523b0 con 0x7f7f500a9d70 2024-03-29T11:36:29.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:29.024 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.024+0000 7f7f4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:29.024 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:29.024 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 10 librados: watch_flush enter 2024-03-29T11:36:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 10 librados: watch_flush exit 2024-03-29T11:36:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:36:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:36:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 >> v1:172.21.15.71:6816/1596514235 conn(0x7f7f3406dd90 legacy=0x7f7f34070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:29.027 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 10 monclient: shutdown 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.027+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 >> v1:172.21.15.71:6790/0 conn(0x7f7f500a9d70 legacy=0x7f7f5006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.028+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 shutdown_connections 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.028+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 >> 172.21.15.71:0/3459675369 conn(0x7f7f50066a00 msgr2=0x7f7f500b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.028+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 shutdown_connections 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.028+0000 7f7f57139640 1 -- 172.21.15.71:0/3459675369 wait complete. 2024-03-29T11:36:29.028 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:29.028+0000 7f7f57139640 1 librados: shutdown 2024-03-29T11:36:29.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:35.846 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:35.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.984+0000 7f14ebfff640 10 monclient: get_monmap_and_config 2024-03-29T11:36:35.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.984+0000 7f14ebfff640 10 monclient: build_initial_monmap 2024-03-29T11:36:35.984 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.984+0000 7f14ebfff640 10 monclient: monmap: 2024-03-29T11:36:35.984 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:35.984 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:35.984 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:35.985095+0000 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:35.985095+0000 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.985+0000 7f14ebfff640 1 Processor -- start 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.985+0000 7f14ebfff640 1 -- start start 2024-03-29T11:36:35.985 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.985+0000 7f14ebfff640 10 monclient: init 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14ebfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14ebfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14ebfff640 10 monclient(hunting): picked mon.noname-b con 0x7f14ec0a9750 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14eaffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f14ec0a9750 0x7f14ec141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55024/0 (socket says 172.21.15.71:55024) 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14eaffd640 1 -- 172.21.15.71:0/3832866134 learned_addr learned my addr 172.21.15.71:0/3832866134 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14ebfff640 10 monclient(hunting): picked mon.noname-c con 0x7f14ec0ae220 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.986+0000 7f14ebfff640 10 monclient(hunting): picked mon.noname-a con 0x7f14ec13b9d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec142490 con 0x7f14ec13b9d0 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec0ac130 con 0x7f14ec0ae220 2024-03-29T11:36:35.988 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec0ac410 con 0x7f14ec0a9750 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14ebfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14ebfff640 10 monclient(hunting): authenticate will time out at 1050.141619s 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276403919 0 0) 0x7f14ec0ac410 con 0x7f14ec0a9750 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14e9ffb640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.987+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d4003520 con 0x7f14ec0a9750 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.988+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4251018807 0 0) 0x7f14d4003520 con 0x7f14ec0a9750 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.988+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 >> v1:172.21.15.81:6789/0 conn(0x7f14ec0ae220 legacy=0x7f14ec13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:35.989 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.988+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 >> v1:172.21.15.71:6789/0 conn(0x7f14ec13b9d0 legacy=0x7f14ec149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.990+0000 7f14e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.990+0000 7f14e9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.990+0000 7f14e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec0afa40 con 0x7f14ec0a9750 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f14e4001a10 con 0x7f14ec0a9750 2024-03-29T11:36:35.992 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: dump: 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14e4001c30 con 0x7f14ec0a9750 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:35.993 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 1 -- 172.21.15.71:0/3832866134 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f14e4002000 con 0x7f14ec0a9750 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14e9ffb640 10 monclient: dump: 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:35.994 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.991+0000 7f14ebfff640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.992+0000 7f14ebfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.992+0000 7f14ebfff640 4 set_mon_vals no callback set 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.992+0000 7f14ebfff640 10 monclient: shutdown 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.992+0000 7f14ebfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.992+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 >> v1:172.21.15.71:6790/0 conn(0x7f14ec0a9750 legacy=0x7f14ec141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.993+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 shutdown_connections 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.993+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 >> 172.21.15.71:0/3832866134 conn(0x7f14ec0673e0 msgr2=0x7f14ec0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.993+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 shutdown_connections 2024-03-29T11:36:35.995 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.993+0000 7f14ebfff640 1 -- 172.21.15.71:0/3832866134 wait complete. 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.994+0000 7f14ebfff640 10 monclient: build_initial_monmap 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.994+0000 7f14ebfff640 10 monclient: monmap: 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:35.998 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.994+0000 7f14ebfff640 1 librados: starting msgr at 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.994+0000 7f14ebfff640 1 librados: starting objecter 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.995+0000 7f14ebfff640 1 Processor -- start 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.995+0000 7f14ebfff640 1 -- start start 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.995+0000 7f14ebfff640 1 librados: setting wanted keys 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.995+0000 7f14ebfff640 1 librados: calling monclient init 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.995+0000 7f14ebfff640 10 monclient: init 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.996+0000 7f14ebfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.996+0000 7f14ebfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 10 monclient(hunting): picked mon.noname-a con 0x7f14ec0a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 10 monclient(hunting): picked mon.noname-b con 0x7f14ec0ae220 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 10 monclient(hunting): picked mon.noname-c con 0x7f14ec13b9d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:35.999 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec13be30 con 0x7f14ec0a9750 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec13cc30 con 0x7f14ec0ae220 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14ec127f90 con 0x7f14ec13b9d0 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ebfff640 10 monclient(hunting): authenticate will time out at 1050.151158s 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ea7fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f14ec0ae220 0x7f14ec13d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:57224/0 (socket says 172.21.15.71:57224) 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.997+0000 7f14ea7fc640 1 -- 172.21.15.71:0/3388448883 learned_addr learned my addr 172.21.15.71:0/3388448883 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 client.?.objecter ms_handle_connect 0x7f14ec0ae220 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 client.?.objecter ms_handle_connect 0x7f14ec13b9d0 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 client.?.objecter ms_handle_connect 0x7f14ec0a9750 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544741600 0 0) 0x7f14ec13be30 con 0x7f14ec0a9750 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.000 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d8004eb0 con 0x7f14ec0a9750 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663640031 0 0) 0x7f14ec127f90 con 0x7f14ec13b9d0 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:36:36.001 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.998+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ec13be30 con 0x7f14ec13b9d0 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503977208 0 0) 0x7f14ec13cc30 con 0x7f14ec0ae220 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ec127f90 con 0x7f14ec0ae220 2024-03-29T11:36:36.006 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1327523450 0 0) 0x7f14ec13be30 con 0x7f14ec13b9d0 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14ec13cc30 con 0x7f14ec13b9d0 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f14e0001670 con 0x7f14ec13b9d0 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): dump: 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:36.007 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3832047474 0 0) 0x7f14d8004eb0 con 0x7f14ec0a9750 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14ec13be30 con 0x7f14ec0a9750 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f14e40018d0 con 0x7f14ec0a9750 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient(hunting): dump: 2024-03-29T11:36:36.008 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3520622332 0 0) 0x7f14ec13cc30 con 0x7f14ec13b9d0 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 >> v1:172.21.15.81:6789/0 conn(0x7f14ec0ae220 legacy=0x7f14ec13d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 >> v1:172.21.15.71:6789/0 conn(0x7f14ec0a9750 legacy=0x7f14ec13c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:36.009 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 monclient: found mon.c 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:35.999+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec128120 con 0x7f14ec13b9d0 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 10 monclient: _renew_subs 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:36.010 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14ec0b2330 con 0x7f14ec13b9d0 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 10 monclient: _renew_subs 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14ec1400d0 con 0x7f14ec13b9d0 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14ebfff640 1 librados: init done 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14e0001b60 con 0x7f14ec13b9d0 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f14e0003790 con 0x7f14ec13b9d0 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14d37fe640 10 monclient: dump: 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:36.011 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.000+0000 7f14e97fa640 4 set_mon_vals no callback set 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14ebfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14ebfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:36.012 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f14e0019760 con 0x7f14ec13b9d0 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc005080 con 0x7f14ec13b9d0 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f14e0053330 con 0x7f14ec13b9d0 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14d37fe640 10 client.4415.objecter ms_dispatch 0x7f14ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14d37fe640 3 client.4415.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14d37fe640 3 client.4415.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.002+0000 7f14d37fe640 20 client.4415.objecter dump_active .. 0 homeless 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.003+0000 7f14eaffd640 10 monclient: get_auth_request con 0x7f14d806df50 auth_method 0 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.003+0000 7f14d37fe640 10 client.4415.objecter ms_handle_connect 0x7f14d806df50 2024-03-29T11:36:36.013 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.010+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14e004f5d0 con 0x7f14ec13b9d0 2024-03-29T11:36:36.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.010+0000 7f14d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:36.014 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.010+0000 7f14d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:36.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.329+0000 7f14ebfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:36.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.329+0000 7f14ebfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:36.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.329+0000 7f14ebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:36.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.329+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14cc005330 con 0x7f14ec13b9d0 2024-03-29T11:36:36.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.330+0000 7f14d37fe640 1 -- 172.21.15.71:0/3388448883 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f14e004f770 con 0x7f14ec13b9d0 2024-03-29T11:36:36.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.330+0000 7f14d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:36.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.330+0000 7f14d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:36.331 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:36.331 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:36.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.333+0000 7f14ebfff640 10 librados: watch_flush enter 2024-03-29T11:36:36.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.335+0000 7f14ebfff640 10 librados: watch_flush exit 2024-03-29T11:36:36.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.335+0000 7f14ebfff640 20 client.4415.objecter shutdown clearing up homeless session... 2024-03-29T11:36:36.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.335+0000 7f14ebfff640 10 client.4415.objecter successfully canceled tick 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 >> v1:172.21.15.71:6816/1596514235 conn(0x7f14d806df50 legacy=0x7f14d8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 10 monclient: shutdown 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 >> v1:172.21.15.71:6790/0 conn(0x7f14ec13b9d0 legacy=0x7f14ec127880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 shutdown_connections 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 >> 172.21.15.71:0/3388448883 conn(0x7f14ec0673e0 msgr2=0x7f14ec069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 shutdown_connections 2024-03-29T11:36:36.336 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 -- 172.21.15.71:0/3388448883 wait complete. 2024-03-29T11:36:36.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:36.336+0000 7f14ebfff640 1 librados: shutdown 2024-03-29T11:36:36.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:43.160 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:43.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.316+0000 7f858684a640 10 monclient: get_monmap_and_config 2024-03-29T11:36:43.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.316+0000 7f858684a640 10 monclient: build_initial_monmap 2024-03-29T11:36:43.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.316+0000 7f858684a640 10 monclient: monmap: 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:43.318133+0000 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:43.318133+0000 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:43.317 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.317+0000 7f858684a640 1 Processor -- start 2024-03-29T11:36:43.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.317+0000 7f858684a640 1 -- start start 2024-03-29T11:36:43.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.317+0000 7f858684a640 10 monclient: init 2024-03-29T11:36:43.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient(hunting): picked mon.noname-a con 0x7f85800b4330 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient(hunting): picked mon.noname-c con 0x7f85801470b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient(hunting): picked mon.noname-b con 0x7f85801439b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858006f2a0 con 0x7f85800b4330 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8580146b90 con 0x7f85801470b0 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85800b2ca0 con 0x7f85801439b0 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:43.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.318+0000 7f858684a640 10 monclient(hunting): authenticate will time out at 1057.473254s 2024-03-29T11:36:43.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.320+0000 7f8584de7640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f85801439b0 0x7f8580151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:42244/0 (socket says 172.21.15.71:42244) 2024-03-29T11:36:43.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.320+0000 7f8584de7640 1 -- 172.21.15.71:0/1665712050 learned_addr learned my addr 172.21.15.71:0/1665712050 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:43.321 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721736754 0 0) 0x7f858006f2a0 con 0x7f85800b4330 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8568003520 con 0x7f85800b4330 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565353145 0 0) 0x7f85800b2ca0 con 0x7f85801439b0 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f858006f2a0 con 0x7f85801439b0 2024-03-29T11:36:43.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2137047834 0 0) 0x7f8568003520 con 0x7f85800b4330 2024-03-29T11:36:43.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 >> v1:172.21.15.71:6790/0 conn(0x7f85801439b0 legacy=0x7f8580151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.321+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 >> v1:172.21.15.81:6789/0 conn(0x7f85801470b0 legacy=0x7f85800b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:43.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 1 monclient: found mon.noname-a 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858006e0c0 con 0x7f85800b4330 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: _finish_auth 0 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f858684a640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f858684a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8578001a10 con 0x7f85800b4330 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:43.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f857effd640 10 monclient: dump: 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.324+0000 7f858684a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8578001c30 con 0x7f85800b4330 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f857effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:43.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f858684a640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f858684a640 4 set_mon_vals no callback set 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f857effd640 1 -- 172.21.15.71:0/1665712050 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8578001fc0 con 0x7f85800b4330 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f857effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f857effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.325+0000 7f857effd640 10 monclient: dump: 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:43.328 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 10 monclient: shutdown 2024-03-29T11:36:43.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:43.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 1 -- 172.21.15.71:0/1665712050 >> v1:172.21.15.71:6789/0 conn(0x7f85800b4330 legacy=0x7f85800b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.329 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 1 -- 172.21.15.71:0/1665712050 shutdown_connections 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 1 -- 172.21.15.71:0/1665712050 >> 172.21.15.71:0/1665712050 conn(0x7f8580066a00 msgr2=0x7f8580066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 1 -- 172.21.15.71:0/1665712050 shutdown_connections 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.326+0000 7f858684a640 1 -- 172.21.15.71:0/1665712050 wait complete. 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.327+0000 7f858684a640 10 monclient: build_initial_monmap 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.327+0000 7f858684a640 10 monclient: monmap: 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.330 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:43.331 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:43.331 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:43.331 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.327+0000 7f858684a640 1 librados: starting msgr at 2024-03-29T11:36:43.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.327+0000 7f858684a640 1 librados: starting objecter 2024-03-29T11:36:43.331 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.327+0000 7f858684a640 1 Processor -- start 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.328+0000 7f858684a640 1 -- start start 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.328+0000 7f858684a640 1 librados: setting wanted keys 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.328+0000 7f858684a640 1 librados: calling monclient init 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.328+0000 7f858684a640 10 monclient: init 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient(hunting): picked mon.noname-c con 0x7f85800b4330 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient(hunting): picked mon.noname-b con 0x7f85801439b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient(hunting): picked mon.noname-a con 0x7f85801470b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8580127330 con 0x7f85801470b0 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8580124ed0 con 0x7f85801439b0 2024-03-29T11:36:43.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85801db760 con 0x7f85800b4330 2024-03-29T11:36:43.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:43.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.329+0000 7f858684a640 10 monclient(hunting): authenticate will time out at 1057.484484s 2024-03-29T11:36:43.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.331+0000 7f857ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f85800b4330 0x7f8580125940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:43642/0 (socket says 172.21.15.71:43642) 2024-03-29T11:36:43.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.331+0000 7f857ffff640 1 -- 172.21.15.71:0/3534596569 learned_addr learned my addr 172.21.15.71:0/3534596569 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:43.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 client.?.objecter ms_handle_connect 0x7f85800b4330 2024-03-29T11:36:43.334 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:43.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 client.?.objecter ms_handle_connect 0x7f85801439b0 2024-03-29T11:36:43.335 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325848712 0 0) 0x7f85801db760 con 0x7f85800b4330 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c004eb0 con 0x7f85800b4330 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980828525 0 0) 0x7f8580124ed0 con 0x7f85801439b0 2024-03-29T11:36:43.341 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85801db760 con 0x7f85801439b0 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 client.?.objecter ms_handle_connect 0x7f85801470b0 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.334+0000 7f857cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023767114 0 0) 0x7f8580127330 con 0x7f85801470b0 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8580124ed0 con 0x7f85801470b0 2024-03-29T11:36:43.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2925921614 0 0) 0x7f8580124ed0 con 0x7f85801470b0 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8580127330 con 0x7f85801470b0 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8570001680 con 0x7f85801470b0 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): dump: 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:43.343 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094652333 0 0) 0x7f856c004eb0 con 0x7f85800b4330 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8580124ed0 con 0x7f85800b4330 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8578001f40 con 0x7f85800b4330 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.335+0000 7f857cff9640 10 monclient(hunting): dump: 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.344 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569833251 0 0) 0x7f85801db760 con 0x7f85801439b0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f856c004eb0 con 0x7f85801439b0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 741697390 0 0) 0x7f8580127330 con 0x7f85801470b0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 >> v1:172.21.15.71:6790/0 conn(0x7f85800b4330 legacy=0x7f8580125940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 >> v1:172.21.15.81:6789/0 conn(0x7f85801439b0 legacy=0x7f85801dbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 monclient: found mon.a 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.336+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8580126050 con 0x7f85801470b0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:43.345 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 10 monclient: _renew_subs 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8580122690 con 0x7f85801470b0 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 10 monclient: _renew_subs 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85801dc390 con 0x7f85801470b0 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f858684a640 1 librados: init done 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85700017e0 con 0x7f85801470b0 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f85700047e0 con 0x7f85801470b0 2024-03-29T11:36:43.346 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.337+0000 7f857e7fc640 4 set_mon_vals no callback set 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.338+0000 7f857cff9640 10 monclient: dump: 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.338+0000 7f858684a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:43.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.339+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8570004940 con 0x7f85801470b0 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.339+0000 7f858684a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.339+0000 7f858684a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.339+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8560005080 con 0x7f85801470b0 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857ffff640 10 monclient: get_auth_request con 0x7f856c06e150 auth_method 0 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f8570084790 con 0x7f85801470b0 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857cff9640 10 client.4392.objecter ms_dispatch 0x7f8580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857cff9640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857cff9640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857cff9640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:36:43.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.340+0000 7f857cff9640 10 client.4392.objecter ms_handle_connect 0x7f856c06e150 2024-03-29T11:36:43.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.348+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8570050070 con 0x7f85801470b0 2024-03-29T11:36:43.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.348+0000 7f857cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:43.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.348+0000 7f857cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:43.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f858684a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:43.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f858684a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:43.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f858684a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:43.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8560005330 con 0x7f85801470b0 2024-03-29T11:36:43.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f857cff9640 1 -- 172.21.15.71:0/3534596569 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f8570051fa0 con 0x7f85801470b0 2024-03-29T11:36:43.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f857cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:43.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.669+0000 7f857cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:43.670 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:43.670 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:43.673 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.672+0000 7f858684a640 10 librados: watch_flush enter 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 10 librados: watch_flush exit 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 >> v1:172.21.15.71:6816/1596514235 conn(0x7f856c06e150 legacy=0x7f856c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 10 monclient: shutdown 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 >> v1:172.21.15.71:6789/0 conn(0x7f85801470b0 legacy=0x7f8580140e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 shutdown_connections 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 >> 172.21.15.71:0/3534596569 conn(0x7f8580066a00 msgr2=0x7f8580070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 shutdown_connections 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 -- 172.21.15.71:0/3534596569 wait complete. 2024-03-29T11:36:43.674 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:43.673+0000 7f858684a640 1 librados: shutdown 2024-03-29T11:36:43.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:50.499 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:50.650 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.651+0000 7ff72b32b640 10 monclient: get_monmap_and_config 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.651+0000 7ff72b32b640 10 monclient: build_initial_monmap 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.651+0000 7ff72b32b640 10 monclient: monmap: 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:50.652226+0000 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:50.652226+0000 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:50.651 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:50.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.655+0000 7ff72b32b640 1 Processor -- start 2024-03-29T11:36:50.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.656+0000 7ff72b32b640 1 -- start start 2024-03-29T11:36:50.655 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.656+0000 7ff72b32b640 10 monclient: init 2024-03-29T11:36:50.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:50.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:50.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient(hunting): picked mon.noname-c con 0x7ff71c006a70 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:50.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient(hunting): picked mon.noname-b con 0x7ff71c02b100 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:50.658 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient(hunting): picked mon.noname-a con 0x7ff71c029c30 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c02a270 con 0x7ff71c029c30 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c02c440 con 0x7ff71c006a70 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0c9440 con 0x7ff71c02b100 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.657+0000 7ff72b32b640 10 monclient(hunting): authenticate will time out at 1064.811447s 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff72ab2a640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff71c029c30 0x7ff71c0d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:32781/0 (socket says 172.21.15.71:32781) 2024-03-29T11:36:50.659 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff72ab2a640 1 -- 172.21.15.71:0/3371145314 learned_addr learned my addr 172.21.15.71:0/3371145314 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255443012 0 0) 0x7ff71c02a270 con 0x7ff71c029c30 2024-03-29T11:36:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff729327640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff729327640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.663 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff729327640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff729327640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.660+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff710003520 con 0x7ff71c029c30 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.663+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613355300 0 0) 0x7ff71c0c9440 con 0x7ff71c02b100 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.663+0000 7ff729327640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.663+0000 7ff729327640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.663+0000 7ff729327640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.663+0000 7ff729327640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:36:50.664 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.663+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff71c02a270 con 0x7ff71c02b100 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012291784 0 0) 0x7ff710003520 con 0x7ff71c029c30 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 >> v1:172.21.15.71:6790/0 conn(0x7ff71c02b100 legacy=0x7ff71c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 >> v1:172.21.15.81:6789/0 conn(0x7ff71c006a70 legacy=0x7ff71c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: _finish_hunting 0 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 monclient: found mon.noname-a 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c0c95d0 con 0x7ff71c029c30 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: _finish_auth 0 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: _check_auth_tickets 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:50.666 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff72404f730 con 0x7ff71c029c30 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: dump: 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff724075f10 con 0x7ff71c029c30 2024-03-29T11:36:50.667 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 1 -- 172.21.15.71:0/3371145314 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff7240758b0 con 0x7ff71c029c30 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.665+0000 7ff729327640 10 monclient: dump: 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:50.668 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.666+0000 7ff72b32b640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.666+0000 7ff72b32b640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.666+0000 7ff72b32b640 4 set_mon_vals no callback set 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.667+0000 7ff72b32b640 10 monclient: shutdown 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.667+0000 7ff72b32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.667+0000 7ff72b32b640 1 -- 172.21.15.71:0/3371145314 >> v1:172.21.15.71:6789/0 conn(0x7ff71c029c30 legacy=0x7ff71c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.667+0000 7ff72b32b640 1 -- 172.21.15.71:0/3371145314 shutdown_connections 2024-03-29T11:36:50.669 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.667+0000 7ff72b32b640 1 -- 172.21.15.71:0/3371145314 >> 172.21.15.71:0/3371145314 conn(0x7ff71c0058f0 msgr2=0x7ff71c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:50.670 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.671+0000 7ff72b32b640 1 -- 172.21.15.71:0/3371145314 shutdown_connections 2024-03-29T11:36:50.671 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.672+0000 7ff72b32b640 1 -- 172.21.15.71:0/3371145314 wait complete. 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.673+0000 7ff72b32b640 10 monclient: build_initial_monmap 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.673+0000 7ff72b32b640 10 monclient: monmap: 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:50.679 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 1 librados: starting msgr at 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 1 librados: starting objecter 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 1 Processor -- start 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 1 -- start start 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 1 librados: setting wanted keys 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 1 librados: calling monclient init 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.674+0000 7ff72b32b640 10 monclient: init 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient(hunting): picked mon.noname-c con 0x7ff71c006a70 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:50.680 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient(hunting): picked mon.noname-b con 0x7ff71c029c30 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient(hunting): picked mon.noname-a con 0x7ff71c186db0 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c1872d0 con 0x7ff71c186db0 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c02f4c0 con 0x7ff71c029c30 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c02f7a0 con 0x7ff71c006a70 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.675+0000 7ff72b32b640 10 monclient(hunting): authenticate will time out at 1064.829450s 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.678+0000 7ff72ab2a640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7ff71c186db0 0x7ff71c02db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:32995/0 (socket says 172.21.15.71:32995) 2024-03-29T11:36:50.681 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.678+0000 7ff72ab2a640 1 -- 172.21.15.71:0/1230018392 learned_addr learned my addr 172.21.15.71:0/1230018392 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.682+0000 7ff71affd640 10 client.?.objecter ms_handle_connect 0x7ff71c186db0 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.682+0000 7ff71affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 client.?.objecter ms_handle_connect 0x7ff71c006a70 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 client.?.objecter ms_handle_connect 0x7ff71c029c30 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457253615 0 0) 0x7ff71c1872d0 con 0x7ff71c186db0 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.686 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff708004eb0 con 0x7ff71c186db0 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522605178 0 0) 0x7ff71c02f7a0 con 0x7ff71c006a70 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff71c1872d0 con 0x7ff71c006a70 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.683+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367064460 0 0) 0x7ff71c02f4c0 con 0x7ff71c029c30 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:36:50.687 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff71c02f7a0 con 0x7ff71c029c30 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 330374993 0 0) 0x7ff71c1872d0 con 0x7ff71c006a70 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff71c02f4c0 con 0x7ff71c006a70 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff714001720 con 0x7ff71c006a70 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): dump: 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:50.688 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2384950880 0 0) 0x7ff708004eb0 con 0x7ff71c186db0 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff71c1872d0 con 0x7ff71c186db0 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.684+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1110803080 0 0) 0x7ff71c02f4c0 con 0x7ff71c006a70 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 >> v1:172.21.15.81:6789/0 conn(0x7ff71c029c30 legacy=0x7ff71c02d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 >> v1:172.21.15.71:6789/0 conn(0x7ff71c186db0 legacy=0x7ff71c02db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 1 monclient: found mon.c 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c191a20 con 0x7ff71c006a70 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 10 monclient: _finish_auth 0 2024-03-29T11:36:50.689 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff71affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 10 monclient: _renew_subs 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c18e9f0 con 0x7ff71c006a70 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 10 monclient: _renew_subs 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c18ee80 con 0x7ff71c006a70 2024-03-29T11:36:50.690 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.685+0000 7ff72b32b640 1 librados: init done 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.687+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff714001c30 con 0x7ff71c006a70 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.687+0000 7ff71affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.687+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff714003790 con 0x7ff71c006a70 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.687+0000 7ff71affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.687+0000 7ff71affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.687+0000 7ff71affd640 10 monclient: dump: 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.692 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.688+0000 7ff72b32b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.688+0000 7ff72b32b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.688+0000 7ff72b32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.688+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c004cd0 con 0x7ff71c006a70 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.688+0000 7ff728b26640 4 set_mon_vals no callback set 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.690+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7ff714019760 con 0x7ff71c006a70 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.691+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7ff714053410 con 0x7ff71c006a70 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.691+0000 7ff71affd640 10 client.4427.objecter ms_dispatch 0x7ff724000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.691+0000 7ff71affd640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:50.693 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.691+0000 7ff71affd640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:50.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.691+0000 7ff71affd640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:36:50.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.691+0000 7ff729b28640 10 monclient: get_auth_request con 0x7ff70806df00 auth_method 0 2024-03-29T11:36:50.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.696+0000 7ff71affd640 10 client.4427.objecter ms_handle_connect 0x7ff70806df00 2024-03-29T11:36:50.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.700+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff71404f6b0 con 0x7ff71c006a70 2024-03-29T11:36:50.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.700+0000 7ff71affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:50.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:50.700+0000 7ff71affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.019+0000 7ff72b32b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.019+0000 7ff72b32b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.019+0000 7ff72b32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.019+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff71c0c6580 con 0x7ff71c006a70 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.022+0000 7ff71affd640 1 -- 172.21.15.71:0/1230018392 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7ff71404f6b0 con 0x7ff71c006a70 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.022+0000 7ff71affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.022+0000 7ff71affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:51.022 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:51.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 10 librados: watch_flush enter 2024-03-29T11:36:51.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 10 librados: watch_flush exit 2024-03-29T11:36:51.025 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 >> v1:172.21.15.71:6816/1596514235 conn(0x7ff70806df00 legacy=0x7ff7080703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 10 monclient: shutdown 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 >> v1:172.21.15.71:6790/0 conn(0x7ff71c006a70 legacy=0x7ff71c1866a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.025+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 shutdown_connections 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.026+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 >> 172.21.15.71:0/1230018392 conn(0x7ff71c0058f0 msgr2=0x7ff71c0c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.026+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 shutdown_connections 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.026+0000 7ff72b32b640 1 -- 172.21.15.71:0/1230018392 wait complete. 2024-03-29T11:36:51.026 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:51.026+0000 7ff72b32b640 1 librados: shutdown 2024-03-29T11:36:51.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:57.850 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:58.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.002+0000 7f915ee4e640 10 monclient: get_monmap_and_config 2024-03-29T11:36:58.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.002+0000 7f915ee4e640 10 monclient: build_initial_monmap 2024-03-29T11:36:58.002 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.002+0000 7f915ee4e640 10 monclient: monmap: 2024-03-29T11:36:58.002 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:58.002 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.002 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:36:58.003516+0000 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:36:58.003516+0000 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.003 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.003+0000 7f915ee4e640 1 Processor -- start 2024-03-29T11:36:58.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.003+0000 7f915ee4e640 1 -- start start 2024-03-29T11:36:58.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.003+0000 7f915ee4e640 10 monclient: init 2024-03-29T11:36:58.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.004+0000 7f915ee4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:58.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.004+0000 7f915ee4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:58.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.004+0000 7f915ee4e640 10 monclient(hunting): picked mon.noname-a con 0x7f91580b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:58.007 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.007+0000 7f915ee4e640 10 monclient(hunting): picked mon.noname-c con 0x7f915814a700 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:58.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.010+0000 7f9157fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f915814a700 0x7f91580abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:58297/0 (socket says 172.21.15.71:58297) 2024-03-29T11:36:58.015 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 10 monclient(hunting): picked mon.noname-b con 0x7f91580ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91580aba70 con 0x7f91580b0770 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91580aecb0 con 0x7f915814a700 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f9157fff640 1 -- 172.21.15.71:0/2384413175 learned_addr learned my addr 172.21.15.71:0/2384413175 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 1 -- 172.21.15.71:0/2384413175 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91580af150 con 0x7f91580ac5a0 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 10 monclient(hunting): authenticate will time out at 1072.165168s 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959214174 0 0) 0x7f91580aecb0 con 0x7f915814a700 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.016 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9148003520 con 0x7f915814a700 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953340381 0 0) 0x7f91580af150 con 0x7f91580ac5a0 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91580aecb0 con 0x7f91580ac5a0 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390406950 0 0) 0x7f91580aba70 con 0x7f91580b0770 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.017 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91580af150 con 0x7f91580b0770 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3265505285 0 0) 0x7f9148003520 con 0x7f915814a700 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 >> v1:172.21.15.71:6790/0 conn(0x7f91580ac5a0 legacy=0x7f9158151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 >> v1:172.21.15.71:6789/0 conn(0x7f91580b0770 legacy=0x7f9158149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 monclient: found mon.noname-c 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91580ac9d0 con 0x7f915814a700 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:58.018 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f91440016b0 con 0x7f915814a700 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f91577fe640 10 monclient: dump: 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.019 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.011+0000 7f915ee4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9144001810 con 0x7f915814a700 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f91577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f91577fe640 1 -- 172.21.15.71:0/2384413175 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9144001c20 con 0x7f915814a700 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f91577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f91577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f91577fe640 10 monclient: dump: 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.020 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f915ee4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.012+0000 7f915ee4e640 4 set_mon_vals no callback set 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.013+0000 7f915ee4e640 10 monclient: shutdown 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.013+0000 7f915ee4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.013+0000 7f915ee4e640 1 -- 172.21.15.71:0/2384413175 >> v1:172.21.15.81:6789/0 conn(0x7f915814a700 legacy=0x7f91580abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.013+0000 7f915ee4e640 1 -- 172.21.15.71:0/2384413175 shutdown_connections 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.013+0000 7f915ee4e640 1 -- 172.21.15.71:0/2384413175 >> 172.21.15.71:0/2384413175 conn(0x7f915806b660 msgr2=0x7f915806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.014+0000 7f915ee4e640 1 -- 172.21.15.71:0/2384413175 shutdown_connections 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.014+0000 7f915ee4e640 1 -- 172.21.15.71:0/2384413175 wait complete. 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.015+0000 7f915ee4e640 10 monclient: build_initial_monmap 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.015+0000 7f915ee4e640 10 monclient: monmap: 2024-03-29T11:36:58.021 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 1 librados: starting msgr at 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 1 librados: starting objecter 2024-03-29T11:36:58.022 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 1 Processor -- start 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 1 -- start start 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 1 librados: setting wanted keys 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 1 librados: calling monclient init 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.016+0000 7f915ee4e640 10 monclient: init 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient(hunting): picked mon.noname-c con 0x7f91580ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient(hunting): picked mon.noname-a con 0x7f91580b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient(hunting): picked mon.noname-b con 0x7f915814a700 addr v1:172.21.15.81:6789/0 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915811e160 con 0x7f91580b0770 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915811de20 con 0x7f915814a700 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915811cf50 con 0x7f91580ac5a0 2024-03-29T11:36:58.031 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.017+0000 7f915ee4e640 10 monclient(hunting): authenticate will time out at 1072.171574s 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.023+0000 7f9157fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f91580b0770 0x7f91581e10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39517/0 (socket says 172.21.15.71:39517) 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.023+0000 7f9157fff640 1 -- 172.21.15.71:0/113215117 learned_addr learned my addr 172.21.15.71:0/113215117 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.024+0000 7f91557fa640 10 client.?.objecter ms_handle_connect 0x7f91580b0770 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.024+0000 7f91557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.024+0000 7f91557fa640 10 client.?.objecter ms_handle_connect 0x7f91580ac5a0 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.024+0000 7f91557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 client.?.objecter ms_handle_connect 0x7f915814a700 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951014368 0 0) 0x7f915811e160 con 0x7f91580b0770 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.032 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913c004eb0 con 0x7f91580b0770 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089334741 0 0) 0x7f915811cf50 con 0x7f91580ac5a0 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915811e160 con 0x7f91580ac5a0 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277345959 0 0) 0x7f915811de20 con 0x7f915814a700 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f915811cf50 con 0x7f915814a700 2024-03-29T11:36:58.033 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3554599116 0 0) 0x7f913c004eb0 con 0x7f91580b0770 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f915811de20 con 0x7f91580b0770 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693878825 0 0) 0x7f915811cf50 con 0x7f915814a700 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f913c004eb0 con 0x7f915814a700 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1076369233 0 0) 0x7f915811e160 con 0x7f91580ac5a0 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f915811cf50 con 0x7f91580ac5a0 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9144001be0 con 0x7f91580b0770 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): dump: 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.034 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9150001c30 con 0x7f915814a700 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): dump: 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.035 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f914c0018b0 con 0x7f91580ac5a0 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.025+0000 7f91557fa640 10 monclient(hunting): dump: 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.036 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 16938427 0 0) 0x7f915811de20 con 0x7f91580b0770 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 >> v1:172.21.15.71:6790/0 conn(0x7f91580ac5a0 legacy=0x7f915811d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 >> v1:172.21.15.81:6789/0 conn(0x7f915814a700 legacy=0x7f91581e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 monclient: found mon.a 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:58.037 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91581e1800 con 0x7f91580b0770 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9144004a90 con 0x7f91580b0770 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9144004da0 con 0x7f91580b0770 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91557fa640 10 monclient: dump: 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:36:58.038 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.026+0000 7f91567fc640 4 set_mon_vals no callback set 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 10 monclient: _renew_subs 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91581e1ab0 con 0x7f91580b0770 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 10 monclient: _renew_subs 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:58.039 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9158070430 con 0x7f91580b0770 2024-03-29T11:36:58.040 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.027+0000 7f915ee4e640 1 librados: init done 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.030+0000 7f915ee4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.030+0000 7f915ee4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.030+0000 7f915ee4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.030+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91580ab560 con 0x7f91580b0770 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.033+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f914401ad70 con 0x7f91580b0770 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.034+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f9144084a80 con 0x7f91580b0770 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.034+0000 7f91557fa640 10 client.4404.objecter ms_dispatch 0x7f9158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.034+0000 7f91557fa640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.034+0000 7f91557fa640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:58.041 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.034+0000 7f91557fa640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-29T11:36:58.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.035+0000 7f915cbea640 10 monclient: get_auth_request con 0x7f913c06e1d0 auth_method 0 2024-03-29T11:36:58.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.038+0000 7f91557fa640 10 client.4404.objecter ms_handle_connect 0x7f913c06e1d0 2024-03-29T11:36:58.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.039+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9144050190 con 0x7f91580b0770 2024-03-29T11:36:58.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.039+0000 7f91557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:58.042 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.039+0000 7f91557fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:58.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.362+0000 7f915ee4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.362+0000 7f915ee4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.362+0000 7f915ee4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.362+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91580b0b50 con 0x7f91580b0770 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.363+0000 7f91557fa640 1 -- 172.21.15.71:0/113215117 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f9144052290 con 0x7f91580b0770 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.363+0000 7f91557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.363+0000 7f91557fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:36:58.363 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.366+0000 7f915ee4e640 10 librados: watch_flush enter 2024-03-29T11:36:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.366+0000 7f915ee4e640 10 librados: watch_flush exit 2024-03-29T11:36:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.366+0000 7f915ee4e640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-29T11:36:58.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.366+0000 7f915ee4e640 10 client.4404.objecter successfully canceled tick 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 >> v1:172.21.15.71:6816/1596514235 conn(0x7f913c06e1d0 legacy=0x7f913c070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 10 monclient: shutdown 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 >> v1:172.21.15.71:6789/0 conn(0x7f91580b0770 legacy=0x7f91581e10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 shutdown_connections 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 >> 172.21.15.71:0/113215117 conn(0x7f915806b660 msgr2=0x7f9158147d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 shutdown_connections 2024-03-29T11:36:58.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.367+0000 7f915ee4e640 1 -- 172.21.15.71:0/113215117 wait complete. 2024-03-29T11:36:58.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:36:58.369+0000 7f915ee4e640 1 librados: shutdown 2024-03-29T11:36:58.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:05.194 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.348+0000 7fe09359e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.348+0000 7fe09359e640 10 monclient: build_initial_monmap 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.348+0000 7fe09359e640 10 monclient: monmap: 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:05.349862+0000 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:05.349862+0000 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:05.350 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:05.351 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:05.351 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:05.351 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:05.351 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:05.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.351+0000 7fe09359e640 1 Processor -- start 2024-03-29T11:37:05.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.351+0000 7fe09359e640 1 -- start start 2024-03-29T11:37:05.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.351+0000 7fe09359e640 10 monclient: init 2024-03-29T11:37:05.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:05.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:05.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient(hunting): picked mon.noname-a con 0x7fe08c034850 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient(hunting): picked mon.noname-b con 0x7fe08c02a640 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient(hunting): picked mon.noname-c con 0x7fe08c0327d0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c02c820 con 0x7fe08c034850 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c02a120 con 0x7fe08c0327d0 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c0c75b0 con 0x7fe08c02a640 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.352+0000 7fe09359e640 10 monclient(hunting): authenticate will time out at 1079.506663s 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.356+0000 7fe091d9b640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe08c02a640 0x7fe08c0320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46053/0 (socket says 172.21.15.71:46053) 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.356+0000 7fe091d9b640 1 -- 172.21.15.71:0/4209403773 learned_addr learned my addr 172.21.15.71:0/4209403773 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.357+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243097802 0 0) 0x7fe08c0c75b0 con 0x7fe08c02a640 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.357+0000 7fe09159a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.357+0000 7fe09159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:05.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.357+0000 7fe09159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.357+0000 7fe09159a640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.357+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe07c003520 con 0x7fe08c02a640 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.358+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2238524426 0 0) 0x7fe07c003520 con 0x7fe08c02a640 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.358+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 >> v1:172.21.15.81:6789/0 conn(0x7fe08c0327d0 legacy=0x7fe08c0cf240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 >> v1:172.21.15.71:6789/0 conn(0x7fe08c034850 legacy=0x7fe08c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 1 monclient: found mon.noname-b 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:05.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c033110 con 0x7fe08c02a640 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: _finish_auth 0 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe09404f730 con 0x7fe08c02a640 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.359+0000 7fe09159a640 10 monclient: dump: 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:05.363 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:05.364 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:05.364 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:05.364 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:05.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.362+0000 7fe09359e640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.362+0000 7fe09359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.364+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe09406a3b0 con 0x7fe08c02a640 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.364+0000 7fe09159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.364+0000 7fe09359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.364+0000 7fe09359e640 4 set_mon_vals no callback set 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.365+0000 7fe09159a640 1 -- 172.21.15.71:0/4209403773 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe094075710 con 0x7fe08c02a640 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.365+0000 7fe09159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.365+0000 7fe09159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.365+0000 7fe09159a640 10 monclient: dump: 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:05.366 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.366+0000 7fe09359e640 10 monclient: shutdown 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.366+0000 7fe09359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.366+0000 7fe09359e640 1 -- 172.21.15.71:0/4209403773 >> v1:172.21.15.71:6790/0 conn(0x7fe08c02a640 legacy=0x7fe08c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.366+0000 7fe09359e640 1 -- 172.21.15.71:0/4209403773 shutdown_connections 2024-03-29T11:37:05.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.366+0000 7fe09359e640 1 -- 172.21.15.71:0/4209403773 >> 172.21.15.71:0/4209403773 conn(0x7fe08c0058f0 msgr2=0x7fe08c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:05.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.368+0000 7fe09359e640 1 -- 172.21.15.71:0/4209403773 shutdown_connections 2024-03-29T11:37:05.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.368+0000 7fe09359e640 1 -- 172.21.15.71:0/4209403773 wait complete. 2024-03-29T11:37:05.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 10 monclient: build_initial_monmap 2024-03-29T11:37:05.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 10 monclient: monmap: 2024-03-29T11:37:05.370 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:05.370 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 1 librados: starting msgr at 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 1 librados: starting objecter 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 1 Processor -- start 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 1 -- start start 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 1 librados: setting wanted keys 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 1 librados: calling monclient init 2024-03-29T11:37:05.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.370+0000 7fe09359e640 10 monclient: init 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient(hunting): picked mon.noname-c con 0x7fe08c02a640 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient(hunting): picked mon.noname-b con 0x7fe08c0327d0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient(hunting): picked mon.noname-a con 0x7fe08c034850 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c02d990 con 0x7fe08c034850 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c02f140 con 0x7fe08c0327d0 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c0c6870 con 0x7fe08c02a640 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:05.373 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.371+0000 7fe09359e640 10 monclient(hunting): authenticate will time out at 1079.525604s 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.374+0000 7fe09259c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe08c02a640 0x7fe08c02cfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46189/0 (socket says 172.21.15.71:46189) 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.374+0000 7fe09259c640 1 -- 172.21.15.71:0/218637322 learned_addr learned my addr 172.21.15.71:0/218637322 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 client.?.objecter ms_handle_connect 0x7fe08c02a640 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 client.?.objecter ms_handle_connect 0x7fe08c034850 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662033450 0 0) 0x7fe08c0c6870 con 0x7fe08c02a640 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:05.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe074004eb0 con 0x7fe08c02a640 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706596400 0 0) 0x7fe08c02d990 con 0x7fe08c034850 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.375+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c0c6870 con 0x7fe08c034850 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3398902692 0 0) 0x7fe08c0c6870 con 0x7fe08c034850 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0740054e0 con 0x7fe08c034850 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe0880016c0 con 0x7fe08c034850 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:05.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1747383543 0 0) 0x7fe0740054e0 con 0x7fe08c034850 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 >> v1:172.21.15.71:6790/0 conn(0x7fe08c02a640 legacy=0x7fe08c02cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.376+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 >> v1:172.21.15.81:6789/0 conn(0x7fe08c0327d0 legacy=0x7fe08c17a630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:05.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 1 monclient: found mon.a 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c0c6a00 con 0x7fe08c034850 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 10 monclient: _renew_subs 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c0c6cb0 con 0x7fe08c034850 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:05.380 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 10 monclient: _renew_subs 2024-03-29T11:37:05.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:05.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c1861d0 con 0x7fe08c034850 2024-03-29T11:37:05.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.377+0000 7fe09359e640 1 librados: init done 2024-03-29T11:37:05.381 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.378+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe088001cb0 con 0x7fe08c034850 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.378+0000 7fe082ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.378+0000 7fe090d99640 4 set_mon_vals no callback set 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe09359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe088003830 con 0x7fe08c034850 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe082ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe082ffd640 10 monclient: dump: 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.382 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fe088019800 con 0x7fe08c034850 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe09359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe09359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.379+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe078005080 con 0x7fe08c034850 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.380+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fe088083640 con 0x7fe08c034850 2024-03-29T11:37:05.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.380+0000 7fe082ffd640 10 client.4407.objecter ms_dispatch 0x7fe094000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:05.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.380+0000 7fe082ffd640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:05.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.380+0000 7fe082ffd640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:05.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.380+0000 7fe082ffd640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:37:05.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.381+0000 7fe09259c640 10 monclient: get_auth_request con 0x7fe07406dee0 auth_method 0 2024-03-29T11:37:05.384 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.381+0000 7fe082ffd640 10 client.4407.objecter ms_handle_connect 0x7fe07406dee0 2024-03-29T11:37:05.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.391+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe08804edd0 con 0x7fe08c034850 2024-03-29T11:37:05.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.391+0000 7fe082ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:05.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.391+0000 7fe082ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:05.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.712+0000 7fe09359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:05.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.712+0000 7fe09359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:05.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.712+0000 7fe09359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:05.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.712+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe078005330 con 0x7fe08c034850 2024-03-29T11:37:05.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.713+0000 7fe082ffd640 1 -- 172.21.15.71:0/218637322 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fe088050ed0 con 0x7fe08c034850 2024-03-29T11:37:05.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.713+0000 7fe082ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:05.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.713+0000 7fe082ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:05.713 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:05.713 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:05.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.716+0000 7fe09359e640 10 librados: watch_flush enter 2024-03-29T11:37:05.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.716+0000 7fe09359e640 10 librados: watch_flush exit 2024-03-29T11:37:05.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.716+0000 7fe09359e640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:37:05.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.716+0000 7fe09359e640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 >> v1:172.21.15.71:6816/1596514235 conn(0x7fe07406dee0 legacy=0x7fe0740703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 10 monclient: shutdown 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 >> v1:172.21.15.71:6789/0 conn(0x7fe08c034850 legacy=0x7fe08c17ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 shutdown_connections 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 >> 172.21.15.71:0/218637322 conn(0x7fe08c0058f0 msgr2=0x7fe08c036a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 shutdown_connections 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 -- 172.21.15.71:0/218637322 wait complete. 2024-03-29T11:37:05.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:05.717+0000 7fe09359e640 1 librados: shutdown 2024-03-29T11:37:05.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:12.541 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:12.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.695+0000 7f2f967b9640 10 monclient: get_monmap_and_config 2024-03-29T11:37:12.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.695+0000 7f2f967b9640 10 monclient: build_initial_monmap 2024-03-29T11:37:12.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.696+0000 7f2f967b9640 10 monclient: monmap: 2024-03-29T11:37:12.697 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:12.696921+0000 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:12.696921+0000 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.697+0000 7f2f967b9640 1 Processor -- start 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.697+0000 7f2f967b9640 1 -- start start 2024-03-29T11:37:12.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.697+0000 7f2f967b9640 10 monclient: init 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f900ab360 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9014fb00 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f90153090 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f900a94e0 con 0x7f2f900ab360 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f900afb60 con 0x7f2f90153090 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f900aab80 con 0x7f2f9014fb00 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:12.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.698+0000 7f2f967b9640 10 monclient(hunting): authenticate will time out at 1086.852245s 2024-03-29T11:37:12.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.701+0000 7f2f94d56640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f2f90153090 0x7f2f9015cbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:38555/0 (socket says 172.21.15.71:38555) 2024-03-29T11:37:12.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.701+0000 7f2f94d56640 1 -- 172.21.15.71:0/3064901828 learned_addr learned my addr 172.21.15.71:0/3064901828 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:12.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.704+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339858262 0 0) 0x7f2f900afb60 con 0x7f2f90153090 2024-03-29T11:37:12.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.704+0000 7f2f8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:12.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.704+0000 7f2f8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:12.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.704+0000 7f2f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:12.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.704+0000 7f2f8effd640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:37:12.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.704+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f78003520 con 0x7f2f90153090 2024-03-29T11:37:12.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3077921119 0 0) 0x7f2f78003520 con 0x7f2f90153090 2024-03-29T11:37:12.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 >> v1:172.21.15.71:6790/0 conn(0x7f2f9014fb00 legacy=0x7f2f90152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:12.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 >> v1:172.21.15.71:6789/0 conn(0x7f2f900ab360 legacy=0x7f2f900ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 monclient: found mon.noname-c 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f901568d0 con 0x7f2f90153090 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: _finish_auth 0 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2f84001690 con 0x7f2f90153090 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: dump: 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:12.705 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f967b9640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:37:12.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f967b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:12.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f84001860 con 0x7f2f90153090 2024-03-29T11:37:12.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:12.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f967b9640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.705+0000 7f2f967b9640 4 set_mon_vals no callback set 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f8effd640 1 -- 172.21.15.71:0/3064901828 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2f84001cb0 con 0x7f2f90153090 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f8effd640 10 monclient: dump: 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:12.708 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f967b9640 10 monclient: shutdown 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f967b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.706+0000 7f2f967b9640 1 -- 172.21.15.71:0/3064901828 >> v1:172.21.15.81:6789/0 conn(0x7f2f90153090 legacy=0x7f2f9015cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.707+0000 7f2f967b9640 1 -- 172.21.15.71:0/3064901828 shutdown_connections 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.707+0000 7f2f967b9640 1 -- 172.21.15.71:0/3064901828 >> 172.21.15.71:0/3064901828 conn(0x7f2f900b35e0 msgr2=0x7f2f900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:12.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.709+0000 7f2f967b9640 1 -- 172.21.15.71:0/3064901828 shutdown_connections 2024-03-29T11:37:12.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.710+0000 7f2f967b9640 1 -- 172.21.15.71:0/3064901828 wait complete. 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.710+0000 7f2f967b9640 10 monclient: build_initial_monmap 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.710+0000 7f2f967b9640 10 monclient: monmap: 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:12.711 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 1 librados: starting msgr at 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 1 librados: starting objecter 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 1 Processor -- start 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 1 -- start start 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 1 librados: setting wanted keys 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 1 librados: calling monclient init 2024-03-29T11:37:12.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.711+0000 7f2f967b9640 10 monclient: init 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient(hunting): picked mon.noname-a con 0x7f2f900ab360 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9014fb00 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient(hunting): picked mon.noname-b con 0x7f2f90153090 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f90154730 con 0x7f2f900ab360 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f901565d0 con 0x7f2f90153090 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f901562f0 con 0x7f2f9014fb00 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:12.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.712+0000 7f2f967b9640 10 monclient(hunting): authenticate will time out at 1086.866795s 2024-03-29T11:37:12.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.717+0000 7f2f8f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2f9014fb00 0x7f2f9008aab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33961/0 (socket says 172.21.15.71:33961) 2024-03-29T11:37:12.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.717+0000 7f2f8f7fe640 1 -- 172.21.15.71:0/423577700 learned_addr learned my addr 172.21.15.71:0/423577700 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:12.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.717+0000 7f2f8cff9640 10 client.?.objecter ms_handle_connect 0x7f2f9014fb00 2024-03-29T11:37:12.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.717+0000 7f2f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:12.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 client.?.objecter ms_handle_connect 0x7f2f900ab360 2024-03-29T11:37:12.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873357679 0 0) 0x7f2f901562f0 con 0x7f2f9014fb00 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f7c004eb0 con 0x7f2f9014fb00 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203926064 0 0) 0x7f2f90154730 con 0x7f2f900ab360 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.718+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f901562f0 con 0x7f2f900ab360 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2110477845 0 0) 0x7f2f901562f0 con 0x7f2f900ab360 2024-03-29T11:37:12.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f90154730 con 0x7f2f900ab360 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2f88001bb0 con 0x7f2f900ab360 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 10 monclient(hunting): dump: 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.719+0000 7f2f8cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:12.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3292414263 0 0) 0x7f2f90154730 con 0x7f2f900ab360 2024-03-29T11:37:12.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 >> v1:172.21.15.71:6790/0 conn(0x7f2f9014fb00 legacy=0x7f2f9008aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:12.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 >> v1:172.21.15.81:6789/0 conn(0x7f2f90153090 legacy=0x7f2f9014de80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:12.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:12.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 1 monclient: found mon.a 2024-03-29T11:37:12.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:12.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f90154250 con 0x7f2f900ab360 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 10 monclient: _renew_subs 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f90154500 con 0x7f2f900ab360 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 10 monclient: _renew_subs 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9014e770 con 0x7f2f900ab360 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.720+0000 7f2f967b9640 1 librados: init done 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.722+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f880020c0 con 0x7f2f900ab360 2024-03-29T11:37:12.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.722+0000 7f2f8cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.723+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2f88003b10 con 0x7f2f900ab360 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.723+0000 7f2f8cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.723+0000 7f2f8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.723+0000 7f2f8cff9640 10 monclient: dump: 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:12.728 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.723+0000 7f2f8dffb640 4 set_mon_vals no callback set 2024-03-29T11:37:12.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.724+0000 7f2f967b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:12.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.724+0000 7f2f967b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:12.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.724+0000 7f2f967b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:12.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.724+0000 7f2f967b9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70005080 con 0x7f2f900ab360 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.727+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f2f88019ae0 con 0x7f2f900ab360 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.728+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f2f88083af0 con 0x7f2f900ab360 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.728+0000 7f2f8cff9640 10 client.4410.objecter ms_dispatch 0x7f2f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.728+0000 7f2f8cff9640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.728+0000 7f2f8cff9640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.728+0000 7f2f8cff9640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.729+0000 7f2f8f7fe640 10 monclient: get_auth_request con 0x7f2f7c06de00 auth_method 0 2024-03-29T11:37:12.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.729+0000 7f2f8cff9640 10 client.4410.objecter ms_handle_connect 0x7f2f7c06de00 2024-03-29T11:37:12.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.738+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f8804f200 con 0x7f2f900ab360 2024-03-29T11:37:12.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.738+0000 7f2f8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:12.738 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:12.738+0000 7f2f8cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:13.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.066+0000 7f2f967b9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:13.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.066+0000 7f2f967b9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:13.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.066+0000 7f2f967b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:13.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.066+0000 7f2f967b9640 1 -- 172.21.15.71:0/423577700 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f70005330 con 0x7f2f900ab360 2024-03-29T11:37:13.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.068+0000 7f2f8cff9640 1 -- 172.21.15.71:0/423577700 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f2f88051300 con 0x7f2f900ab360 2024-03-29T11:37:13.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.068+0000 7f2f8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:13.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.068+0000 7f2f8cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:13.068 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:13.068 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:13.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.071+0000 7f2f767fc640 10 librados: watch_flush enter 2024-03-29T11:37:13.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 10 librados: watch_flush exit 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 1 -- 172.21.15.71:0/423577700 >> v1:172.21.15.71:6816/1596514235 conn(0x7f2f7c06de00 legacy=0x7f2f7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 10 monclient: shutdown 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.074+0000 7f2f767fc640 1 -- 172.21.15.71:0/423577700 >> v1:172.21.15.71:6789/0 conn(0x7f2f900ab360 legacy=0x7f2f90071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:13.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.075+0000 7f2f767fc640 1 -- 172.21.15.71:0/423577700 shutdown_connections 2024-03-29T11:37:13.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.075+0000 7f2f767fc640 1 -- 172.21.15.71:0/423577700 >> 172.21.15.71:0/423577700 conn(0x7f2f900b35e0 msgr2=0x7f2f900a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:13.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.076+0000 7f2f767fc640 1 -- 172.21.15.71:0/423577700 shutdown_connections 2024-03-29T11:37:13.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.076+0000 7f2f767fc640 1 -- 172.21.15.71:0/423577700 wait complete. 2024-03-29T11:37:13.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:13.076+0000 7f2f767fc640 1 librados: shutdown 2024-03-29T11:37:13.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:19.903 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:20.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.051+0000 7f454141e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:20.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.051+0000 7f454141e640 10 monclient: build_initial_monmap 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.051+0000 7f454141e640 10 monclient: monmap: 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:20.052709+0000 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:20.052709+0000 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:20.052 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.052+0000 7f454141e640 1 Processor -- start 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.052+0000 7f454141e640 1 -- start start 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.052+0000 7f454141e640 10 monclient: init 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient(hunting): picked mon.noname-c con 0x7f453c1457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient(hunting): picked mon.noname-b con 0x7f453c13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient(hunting): picked mon.noname-a con 0x7f453c0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c0aaef0 con 0x7f453c0a9d70 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c145bd0 con 0x7f453c1457f0 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c145eb0 con 0x7f453c13f650 2024-03-29T11:37:20.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f454141e640 10 monclient(hunting): authenticate will time out at 1094.207713s 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f453b7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f453c0a9d70 0x7f453c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42472/0 (socket says 172.21.15.71:42472) 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.053+0000 7f453b7fe640 1 -- 172.21.15.71:0/936262019 learned_addr learned my addr 172.21.15.71:0/936262019 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.054+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667534599 0 0) 0x7f453c0aaef0 con 0x7f453c0a9d70 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.054+0000 7f4539ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.054+0000 7f4539ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.054+0000 7f4539ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.054+0000 7f4539ffb640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:37:20.055 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.054+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4524003520 con 0x7f453c0a9d70 2024-03-29T11:37:20.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.055+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4266765853 0 0) 0x7f4524003520 con 0x7f453c0a9d70 2024-03-29T11:37:20.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.055+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 >> v1:172.21.15.71:6790/0 conn(0x7f453c13f650 legacy=0x7f453c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:20.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.056+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 >> v1:172.21.15.81:6789/0 conn(0x7f453c1457f0 legacy=0x7f453c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:20.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:20.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 1 monclient: found mon.noname-a 2024-03-29T11:37:20.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:20.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:37:20.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c146340 con 0x7f453c0a9d70 2024-03-29T11:37:20.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f452c001690 con 0x7f453c0a9d70 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f4539ffb640 10 monclient: dump: 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:20.061 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f454141e640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.058+0000 7f454141e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.059+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f452c001860 con 0x7f453c0a9d70 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.059+0000 7f4539ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.059+0000 7f4539ffb640 1 -- 172.21.15.71:0/936262019 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f452c001cb0 con 0x7f453c0a9d70 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.059+0000 7f454141e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.059+0000 7f454141e640 4 set_mon_vals no callback set 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.060+0000 7f4539ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.060+0000 7f4539ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.060+0000 7f4539ffb640 10 monclient: dump: 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:20.062 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.060+0000 7f454141e640 10 monclient: shutdown 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.060+0000 7f454141e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:20.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.060+0000 7f454141e640 1 -- 172.21.15.71:0/936262019 >> v1:172.21.15.71:6789/0 conn(0x7f453c0a9d70 legacy=0x7f453c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.062+0000 7f454141e640 1 -- 172.21.15.71:0/936262019 shutdown_connections 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.062+0000 7f454141e640 1 -- 172.21.15.71:0/936262019 >> 172.21.15.71:0/936262019 conn(0x7f453c066a00 msgr2=0x7f453c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.062+0000 7f454141e640 1 -- 172.21.15.71:0/936262019 shutdown_connections 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.062+0000 7f454141e640 1 -- 172.21.15.71:0/936262019 wait complete. 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.063+0000 7f454141e640 10 monclient: build_initial_monmap 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.063+0000 7f454141e640 10 monclient: monmap: 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:20.066 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.063+0000 7f454141e640 1 librados: starting msgr at 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.063+0000 7f454141e640 1 librados: starting objecter 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 1 Processor -- start 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 1 -- start start 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 1 librados: setting wanted keys 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 1 librados: calling monclient init 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 10 monclient: init 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.064+0000 7f454141e640 10 monclient(hunting): picked mon.noname-b con 0x7f453c0a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 10 monclient(hunting): picked mon.noname-a con 0x7f453c13f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 10 monclient(hunting): picked mon.noname-c con 0x7f453c1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:20.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c1da4b0 con 0x7f453c13f650 2024-03-29T11:37:20.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c142900 con 0x7f453c0a9d70 2024-03-29T11:37:20.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f453c140bd0 con 0x7f453c1457f0 2024-03-29T11:37:20.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:20.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.065+0000 7f454141e640 10 monclient(hunting): authenticate will time out at 1094.219094s 2024-03-29T11:37:20.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.067+0000 7f453a7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f453c13f650 0x7f453c1da970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:43788/0 (socket says 172.21.15.71:43788) 2024-03-29T11:37:20.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.067+0000 7f453a7fc640 1 -- 172.21.15.71:0/459150855 learned_addr learned my addr 172.21.15.71:0/459150855 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:20.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.067+0000 7f45237fe640 10 client.?.objecter ms_handle_connect 0x7f453c13f650 2024-03-29T11:37:20.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.067+0000 7f45237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:20.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353073598 0 0) 0x7f453c1da4b0 con 0x7f453c13f650 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4528004eb0 con 0x7f453c13f650 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 client.?.objecter ms_handle_connect 0x7f453c0a9d70 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095062601 0 0) 0x7f453c142900 con 0x7f453c0a9d70 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:37:20.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c1da4b0 con 0x7f453c0a9d70 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 client.?.objecter ms_handle_connect 0x7f453c1457f0 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295915693 0 0) 0x7f453c140bd0 con 0x7f453c1457f0 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f453c142900 con 0x7f453c1457f0 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3924843254 0 0) 0x7f4528004eb0 con 0x7f453c13f650 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f453c140bd0 con 0x7f453c13f650 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4530001960 con 0x7f453c13f650 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): dump: 2024-03-29T11:37:20.072 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.068+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811798171 0 0) 0x7f453c1da4b0 con 0x7f453c0a9d70 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4528004eb0 con 0x7f453c0a9d70 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4534001a90 con 0x7f453c0a9d70 2024-03-29T11:37:20.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient(hunting): dump: 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1458622835 0 0) 0x7f453c140bd0 con 0x7f453c13f650 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 >> v1:172.21.15.71:6790/0 conn(0x7f453c1457f0 legacy=0x7f453c13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 >> v1:172.21.15.81:6789/0 conn(0x7f453c0a9d70 legacy=0x7f453c1421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 monclient: found mon.a 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c140d60 con 0x7f453c13f650 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f45237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 10 monclient: _renew_subs 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c1db080 con 0x7f453c13f650 2024-03-29T11:37:20.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 10 monclient: _renew_subs 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c13e210 con 0x7f453c13f650 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.069+0000 7f454141e640 1 librados: init done 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45300035f0 con 0x7f453c13f650 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4530003940 con 0x7f453c13f650 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45237fe640 10 monclient: dump: 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.076 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:20.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.070+0000 7f45397fa640 4 set_mon_vals no callback set 2024-03-29T11:37:20.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.071+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f4530003aa0 con 0x7f453c13f650 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.072+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f4530084a30 con 0x7f453c13f650 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.072+0000 7f453affd640 10 monclient: get_auth_request con 0x7f4528076810 auth_method 0 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.072+0000 7f454141e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.072+0000 7f454141e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.072+0000 7f454141e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.072+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c005080 con 0x7f453c13f650 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.073+0000 7f45237fe640 10 client.4416.objecter ms_dispatch 0x7f453c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.073+0000 7f45237fe640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.073+0000 7f45237fe640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.073+0000 7f45237fe640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:37:20.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.073+0000 7f45237fe640 10 client.4416.objecter ms_handle_connect 0x7f4528076810 2024-03-29T11:37:20.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.082+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4530050310 con 0x7f453c13f650 2024-03-29T11:37:20.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.082+0000 7f45237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:20.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.082+0000 7f45237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.397+0000 7f454141e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.397+0000 7f454141e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.397+0000 7f454141e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.397+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f451c005330 con 0x7f453c13f650 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.399+0000 7f45237fe640 1 -- 172.21.15.71:0/459150855 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f4530052240 con 0x7f453c13f650 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.399+0000 7f45237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:20.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.399+0000 7f45237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:20.400 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:20.400 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:20.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.403+0000 7f454141e640 10 librados: watch_flush enter 2024-03-29T11:37:20.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.403+0000 7f454141e640 10 librados: watch_flush exit 2024-03-29T11:37:20.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.403+0000 7f454141e640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:37:20.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.403+0000 7f454141e640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:37:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 >> v1:172.21.15.71:6816/1596514235 conn(0x7f4528076810 legacy=0x7f4528078cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 10 monclient: shutdown 2024-03-29T11:37:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 >> v1:172.21.15.71:6789/0 conn(0x7f453c13f650 legacy=0x7f453c1da970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 shutdown_connections 2024-03-29T11:37:20.404 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 >> 172.21.15.71:0/459150855 conn(0x7f453c066a00 msgr2=0x7f453c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.404+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 shutdown_connections 2024-03-29T11:37:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.405+0000 7f454141e640 1 -- 172.21.15.71:0/459150855 wait complete. 2024-03-29T11:37:20.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:20.405+0000 7f454141e640 1 librados: shutdown 2024-03-29T11:37:20.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:27.227 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:27.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.387+0000 7f5e367cd640 10 monclient: get_monmap_and_config 2024-03-29T11:37:27.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.387+0000 7f5e367cd640 10 monclient: build_initial_monmap 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.387+0000 7f5e367cd640 10 monclient: monmap: 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:27.388409+0000 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:27.388409+0000 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:27.388 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.388+0000 7f5e367cd640 1 Processor -- start 2024-03-29T11:37:27.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.388+0000 7f5e367cd640 1 -- start start 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.388+0000 7f5e367cd640 10 monclient: init 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient(hunting): picked mon.noname-c con 0x7f5e301422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient(hunting): picked mon.noname-a con 0x7f5e30142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient(hunting): picked mon.noname-b con 0x7f5e300b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e30142c80 con 0x7f5e30142690 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e300aae40 con 0x7f5e301422b0 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3013fbd0 con 0x7f5e300b17d0 2024-03-29T11:37:27.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:27.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e367cd640 10 monclient(hunting): authenticate will time out at 1101.543848s 2024-03-29T11:37:27.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e2f7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5e30142690 0x7f5e300b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35836/0 (socket says 172.21.15.71:35836) 2024-03-29T11:37:27.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.389+0000 7f5e2f7fe640 1 -- 172.21.15.71:0/429576027 learned_addr learned my addr 172.21.15.71:0/429576027 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.390+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 367872254 0 0) 0x7f5e300aae40 con 0x7f5e301422b0 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.390+0000 7f5e2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.390+0000 7f5e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.390+0000 7f5e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.390+0000 7f5e2effd640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.390+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e18003520 con 0x7f5e301422b0 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515646421 0 0) 0x7f5e30142c80 con 0x7f5e30142690 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e300aae40 con 0x7f5e30142690 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3084459892 0 0) 0x7f5e300aae40 con 0x7f5e30142690 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 >> v1:172.21.15.71:6790/0 conn(0x7f5e300b17d0 legacy=0x7f5e300b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 >> v1:172.21.15.81:6789/0 conn(0x7f5e301422b0 legacy=0x7f5e3013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 monclient: found mon.noname-a 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:37:27.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e30151940 con 0x7f5e30142690 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: _finish_auth 0 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5e24001670 con 0x7f5e30142690 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: dump: 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.395 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e367cd640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e367cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e24001890 con 0x7f5e30142690 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 1 -- 172.21.15.71:0/429576027 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5e24001ba0 con 0x7f5e30142690 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.391+0000 7f5e2effd640 10 monclient: dump: 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.392+0000 7f5e367cd640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.392+0000 7f5e367cd640 4 set_mon_vals no callback set 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.393+0000 7f5e367cd640 10 monclient: shutdown 2024-03-29T11:37:27.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.393+0000 7f5e367cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:27.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.393+0000 7f5e367cd640 1 -- 172.21.15.71:0/429576027 >> v1:172.21.15.71:6789/0 conn(0x7f5e30142690 legacy=0x7f5e300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.393+0000 7f5e367cd640 1 -- 172.21.15.71:0/429576027 shutdown_connections 2024-03-29T11:37:27.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.393+0000 7f5e367cd640 1 -- 172.21.15.71:0/429576027 >> 172.21.15.71:0/429576027 conn(0x7f5e30066a00 msgr2=0x7f5e30066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:27.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.394+0000 7f5e367cd640 1 -- 172.21.15.71:0/429576027 shutdown_connections 2024-03-29T11:37:27.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.394+0000 7f5e367cd640 1 -- 172.21.15.71:0/429576027 wait complete. 2024-03-29T11:37:27.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 10 monclient: build_initial_monmap 2024-03-29T11:37:27.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 10 monclient: monmap: 2024-03-29T11:37:27.406 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:27.406 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 1 librados: starting msgr at 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 1 librados: starting objecter 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 1 Processor -- start 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 1 -- start start 2024-03-29T11:37:27.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 1 librados: setting wanted keys 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 1 librados: calling monclient init 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.395+0000 7f5e367cd640 10 monclient: init 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.396+0000 7f5e367cd640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.396+0000 7f5e367cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.396+0000 7f5e367cd640 10 monclient(hunting): picked mon.noname-b con 0x7f5e300b17d0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f5e300b17d0 0x7f5e301df540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:39032/0 (socket says 172.21.15.71:39032) 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e367cd640 10 monclient(hunting): picked mon.noname-a con 0x7f5e301422b0 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e367cd640 10 monclient(hunting): picked mon.noname-c con 0x7f5e30142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2ffff640 1 -- 172.21.15.71:0/3243925852 learned_addr learned my addr 172.21.15.71:0/3243925852 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e301dfc50 con 0x7f5e301422b0 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2cff9640 10 client.?.objecter ms_handle_connect 0x7f5e300b17d0 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2cff9640 10 client.?.objecter ms_handle_connect 0x7f5e30142690 2024-03-29T11:37:27.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2cff9640 10 client.?.objecter ms_handle_connect 0x7f5e301422b0 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.397+0000 7f5e2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3006ed40 con 0x7f5e300b17d0 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3006de90 con 0x7f5e30142690 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e367cd640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e367cd640 10 monclient(hunting): authenticate will time out at 1101.552133s 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561945874 0 0) 0x7f5e3006de90 con 0x7f5e30142690 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c004eb0 con 0x7f5e30142690 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376329197 0 0) 0x7f5e3006ed40 con 0x7f5e300b17d0 2024-03-29T11:37:27.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3006de90 con 0x7f5e300b17d0 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614956887 0 0) 0x7f5e301dfc50 con 0x7f5e301422b0 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.398+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3006ed40 con 0x7f5e301422b0 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163968617 0 0) 0x7f5e1c004eb0 con 0x7f5e30142690 2024-03-29T11:37:27.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e301dfc50 con 0x7f5e30142690 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5e28001bc0 con 0x7f5e30142690 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): dump: 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4064032656 0 0) 0x7f5e3006de90 con 0x7f5e300b17d0 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e1c004eb0 con 0x7f5e300b17d0 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5e200019f0 con 0x7f5e300b17d0 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.399+0000 7f5e2cff9640 10 monclient(hunting): dump: 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.412 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3219302875 0 0) 0x7f5e3006ed40 con 0x7f5e301422b0 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e3006de90 con 0x7f5e301422b0 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3434341449 0 0) 0x7f5e1c004eb0 con 0x7f5e300b17d0 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 >> v1:172.21.15.71:6790/0 conn(0x7f5e30142690 legacy=0x7f5e301e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 >> v1:172.21.15.71:6789/0 conn(0x7f5e301422b0 legacy=0x7f5e30070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:27.413 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 monclient: found mon.b 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e301df020 con 0x7f5e300b17d0 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 10 monclient: _renew_subs 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e30071490 con 0x7f5e300b17d0 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:27.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 10 monclient: _renew_subs 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3013ba10 con 0x7f5e300b17d0 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e367cd640 1 librados: init done 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e200035f0 con 0x7f5e300b17d0 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5e20003940 con 0x7f5e300b17d0 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:27.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2cff9640 10 monclient: dump: 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.400+0000 7f5e2e7fc640 4 set_mon_vals no callback set 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.401+0000 7f5e367cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.402+0000 7f5e367cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:27.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.402+0000 7f5e367cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.402+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e10005080 con 0x7f5e300b17d0 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.402+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f5e20003aa0 con 0x7f5e300b17d0 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2f7fe640 10 monclient: get_auth_request con 0x7f5e1c06e180 auth_method 0 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f5e200849f0 con 0x7f5e300b17d0 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2cff9640 10 client.4423.objecter ms_dispatch 0x7f5e30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2cff9640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2cff9640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2cff9640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:37:27.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.403+0000 7f5e2cff9640 10 client.4423.objecter ms_handle_connect 0x7f5e1c06e180 2024-03-29T11:37:27.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.408+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e20050100 con 0x7f5e300b17d0 2024-03-29T11:37:27.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.409+0000 7f5e2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:27.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.409+0000 7f5e2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:27.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e367cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:27.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e367cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:27.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e367cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:27.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e10005330 con 0x7f5e300b17d0 2024-03-29T11:37:27.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e2cff9640 1 -- 172.21.15.71:0/3243925852 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f5e20052200 con 0x7f5e300b17d0 2024-03-29T11:37:27.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:27.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.796+0000 7f5e2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:27.797 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:27.797 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 10 librados: watch_flush enter 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 10 librados: watch_flush exit 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 >> v1:172.21.15.71:6816/1596514235 conn(0x7f5e1c06e180 legacy=0x7f5e1c070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 10 monclient: shutdown 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 >> v1:172.21.15.81:6789/0 conn(0x7f5e300b17d0 legacy=0x7f5e301df540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 shutdown_connections 2024-03-29T11:37:27.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 >> 172.21.15.71:0/3243925852 conn(0x7f5e30066a00 msgr2=0x7f5e3013d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:27.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.800+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 shutdown_connections 2024-03-29T11:37:27.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.801+0000 7f5e367cd640 1 -- 172.21.15.71:0/3243925852 wait complete. 2024-03-29T11:37:27.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:27.801+0000 7f5e367cd640 1 librados: shutdown 2024-03-29T11:37:27.814 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:34.621 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.753+0000 7fa5e3784640 10 monclient: get_monmap_and_config 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.753+0000 7fa5e3784640 10 monclient: build_initial_monmap 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.753+0000 7fa5e3784640 10 monclient: monmap: 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:34.754277+0000 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:34.754277+0000 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:34.753 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:34.754 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:34.754 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:34.754 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:34.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.754+0000 7fa5e3784640 1 Processor -- start 2024-03-29T11:37:34.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.754+0000 7fa5e3784640 1 -- start start 2024-03-29T11:37:34.755 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.754+0000 7fa5e3784640 10 monclient: init 2024-03-29T11:37:34.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:34.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:34.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient(hunting): picked mon.noname-c con 0x7fa5dc1457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:34.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient(hunting): picked mon.noname-b con 0x7fa5dc13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:34.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient(hunting): picked mon.noname-a con 0x7fa5dc0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:34.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc0aaef0 con 0x7fa5dc0a9d70 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc145bd0 con 0x7fa5dc1457f0 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc145eb0 con 0x7fa5dc13f650 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e3784640 10 monclient(hunting): authenticate will time out at 1108.909313s 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e1520640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa5dc1457f0 0x7fa5dc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:58654/0 (socket says 172.21.15.71:58654) 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5e1520640 1 -- 172.21.15.71:0/3139885122 learned_addr learned my addr 172.21.15.71:0/3139885122 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926342474 0 0) 0x7fa5dc145eb0 con 0x7fa5dc13f650 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5d3fff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:37:34.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.755+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c8003520 con 0x7fa5dc13f650 2024-03-29T11:37:34.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.756+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3006549580 0 0) 0x7fa5c8003520 con 0x7fa5dc13f650 2024-03-29T11:37:34.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.756+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 >> v1:172.21.15.81:6789/0 conn(0x7fa5dc1457f0 legacy=0x7fa5dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:34.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.756+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 >> v1:172.21.15.71:6789/0 conn(0x7fa5dc0a9d70 legacy=0x7fa5dc14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 1 monclient: found mon.noname-b 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc146340 con 0x7fa5dc13f650 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa5cc001670 con 0x7fa5dc13f650 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: dump: 2024-03-29T11:37:34.760 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5e3784640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:37:34.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5e3784640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5cc001900 con 0x7fa5dc13f650 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 1 -- 172.21.15.71:0/3139885122 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa5cc001cd0 con 0x7fa5dc13f650 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5d3fff640 10 monclient: dump: 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.762 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5e3784640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.758+0000 7fa5e3784640 4 set_mon_vals no callback set 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.759+0000 7fa5e3784640 10 monclient: shutdown 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.759+0000 7fa5e3784640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.759+0000 7fa5e3784640 1 -- 172.21.15.71:0/3139885122 >> v1:172.21.15.71:6790/0 conn(0x7fa5dc13f650 legacy=0x7fa5dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.759+0000 7fa5e3784640 1 -- 172.21.15.71:0/3139885122 shutdown_connections 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.759+0000 7fa5e3784640 1 -- 172.21.15.71:0/3139885122 >> 172.21.15.71:0/3139885122 conn(0x7fa5dc066a00 msgr2=0x7fa5dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.759+0000 7fa5e3784640 1 -- 172.21.15.71:0/3139885122 shutdown_connections 2024-03-29T11:37:34.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.760+0000 7fa5e3784640 1 -- 172.21.15.71:0/3139885122 wait complete. 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.761+0000 7fa5e3784640 10 monclient: build_initial_monmap 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.761+0000 7fa5e3784640 10 monclient: monmap: 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:34.769 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.761+0000 7fa5e3784640 1 librados: starting msgr at 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.761+0000 7fa5e3784640 1 librados: starting objecter 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.761+0000 7fa5e3784640 1 Processor -- start 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.762+0000 7fa5e3784640 1 -- start start 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.762+0000 7fa5e3784640 1 librados: setting wanted keys 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.762+0000 7fa5e3784640 1 librados: calling monclient init 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.762+0000 7fa5e3784640 10 monclient: init 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.763+0000 7fa5e3784640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:34.770 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.763+0000 7fa5e3784640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.763+0000 7fa5e3784640 10 monclient(hunting): picked mon.noname-a con 0x7fa5dc0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 10 monclient(hunting): picked mon.noname-c con 0x7fa5dc1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 10 monclient(hunting): picked mon.noname-b con 0x7fa5dc1daa90 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc13ba90 con 0x7fa5dc0a9d70 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc13acd0 con 0x7fa5dc1daa90 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e0d1f640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fa5dc1457f0 0x7fa5dc134e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58854/0 (socket says 172.21.15.71:58854) 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e0d1f640 1 -- 172.21.15.71:0/1997420462 learned_addr learned my addr 172.21.15.71:0/1997420462 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5dc13af20 con 0x7fa5dc1457f0 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 client.?.objecter ms_handle_connect 0x7fa5dc1457f0 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5e3784640 10 monclient(hunting): authenticate will time out at 1108.918357s 2024-03-29T11:37:34.771 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260975042 0 0) 0x7fa5dc13af20 con 0x7fa5dc1457f0 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5c0004eb0 con 0x7fa5dc1457f0 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418872736 0 0) 0x7fa5c0004eb0 con 0x7fa5dc1457f0 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5dc13af20 con 0x7fa5dc1457f0 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa5cc001630 con 0x7fa5dc1457f0 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:34.772 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 client.?.objecter ms_handle_connect 0x7fa5dc1daa90 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.764+0000 7fa5d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:34.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.765+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1270736918 0 0) 0x7fa5dc13af20 con 0x7fa5dc1457f0 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.765+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 >> v1:172.21.15.81:6789/0 conn(0x7fa5dc1daa90 legacy=0x7fa5dc135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.765+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 >> v1:172.21.15.71:6789/0 conn(0x7fa5dc0a9d70 legacy=0x7fa5dc1db080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 1 monclient: found mon.c 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5dc13b200 con 0x7fa5dc1457f0 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 10 monclient: _renew_subs 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:34.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5dc135c30 con 0x7fa5dc1457f0 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5cc003960 con 0x7fa5dc1457f0 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fa5cc003db0 con 0x7fa5dc1457f0 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d1ffb640 10 monclient: dump: 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:34.775 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 10 monclient: _renew_subs 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5dc139f00 con 0x7fa5dc1457f0 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5e3784640 1 librados: init done 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.766+0000 7fa5d2ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.768+0000 7fa5e3784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.768+0000 7fa5e3784640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.768+0000 7fa5e3784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.768+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5dc13bc40 con 0x7fa5dc1457f0 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.768+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fa5cc003f10 con 0x7fa5dc1457f0 2024-03-29T11:37:34.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fa5cc084bf0 con 0x7fa5dc1457f0 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5d1ffb640 10 client.4457.objecter ms_dispatch 0x7fa5dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5d1ffb640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5d1ffb640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5e1520640 10 monclient: get_auth_request con 0x7fa5c006db30 auth_method 0 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5d1ffb640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.769+0000 7fa5d1ffb640 10 client.4457.objecter ms_handle_connect 0x7fa5c006db30 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.775+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5dc13bc40 con 0x7fa5dc1457f0 2024-03-29T11:37:34.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.775+0000 7fa5d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:34.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:34.775+0000 7fa5d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.078+0000 7fa5e3784640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.078+0000 7fa5e3784640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.078+0000 7fa5e3784640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:35.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.078+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5dc13bc40 con 0x7fa5dc1457f0 2024-03-29T11:37:35.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.079+0000 7fa5d1ffb640 1 -- 172.21.15.71:0/1997420462 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fa5dc13bc40 con 0x7fa5dc1457f0 2024-03-29T11:37:35.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.079+0000 7fa5d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:35.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.079+0000 7fa5d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:35.079 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:35.079 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:35.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 10 librados: watch_flush enter 2024-03-29T11:37:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 10 librados: watch_flush exit 2024-03-29T11:37:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:37:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:37:35.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 >> v1:172.21.15.71:6816/1596514235 conn(0x7fa5c006db30 legacy=0x7fa5c006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 10 monclient: shutdown 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.082+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 >> v1:172.21.15.71:6790/0 conn(0x7fa5dc1457f0 legacy=0x7fa5dc134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.083+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 shutdown_connections 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.083+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 >> 172.21.15.71:0/1997420462 conn(0x7fa5dc066a00 msgr2=0x7fa5dc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.083+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 shutdown_connections 2024-03-29T11:37:35.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.083+0000 7fa5e3784640 1 -- 172.21.15.71:0/1997420462 wait complete. 2024-03-29T11:37:35.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:35.083+0000 7fa5e3784640 1 librados: shutdown 2024-03-29T11:37:35.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:41.906 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:42.048 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.047+0000 7f4250110640 10 monclient: get_monmap_and_config 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.047+0000 7f4250110640 10 monclient: build_initial_monmap 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.047+0000 7f4250110640 10 monclient: monmap: 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:42.049822+0000 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:42.049822+0000 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:42.049 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:42.050 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:42.050 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:42.050 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:42.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.048+0000 7f4250110640 1 Processor -- start 2024-03-29T11:37:42.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.048+0000 7f4250110640 1 -- start start 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.048+0000 7f4250110640 10 monclient: init 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient(hunting): picked mon.noname-b con 0x7f42481457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient(hunting): picked mon.noname-a con 0x7f424813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient(hunting): picked mon.noname-c con 0x7f42480a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42480aaef0 con 0x7f424813f650 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4248145bd0 con 0x7f42480a9d70 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4248145eb0 con 0x7f42481457f0 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f4250110640 10 monclient(hunting): authenticate will time out at 1116.204832s 2024-03-29T11:37:42.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f424e6ad640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f42480a9d70 0x7f424814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:59756/0 (socket says 172.21.15.71:59756) 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f424e6ad640 1 -- 172.21.15.71:0/292342237 learned_addr learned my addr 172.21.15.71:0/292342237 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.049+0000 7f424deac640 1 --1- 172.21.15.71:0/292342237 >> v1:172.21.15.71:6790/0 conn(0x7f42481457f0 0x7f424813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:58592/0 (socket says 172.21.15.71:58592) 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819374774 0 0) 0x7f4248145bd0 con 0x7f42480a9d70 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234003520 con 0x7f42480a9d70 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 762686795 0 0) 0x7f4234003520 con 0x7f42480a9d70 2024-03-29T11:37:42.053 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 >> v1:172.21.15.71:6790/0 conn(0x7f42481457f0 legacy=0x7f424813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.054 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.050+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 >> v1:172.21.15.71:6789/0 conn(0x7f424813f650 legacy=0x7f42480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 1 monclient: found mon.noname-c 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248146340 con 0x7f42480a9d70 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: _finish_auth 0 2024-03-29T11:37:42.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4240001a10 con 0x7f42480a9d70 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f424ceaa640 10 monclient: dump: 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:42.059 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f4250110640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.053+0000 7f4250110640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4240001be0 con 0x7f42480a9d70 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f424ceaa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f424ceaa640 1 -- 172.21.15.71:0/292342237 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4240002030 con 0x7f42480a9d70 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f424ceaa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f424ceaa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f424ceaa640 10 monclient: dump: 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:42.060 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f4250110640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.054+0000 7f4250110640 4 set_mon_vals no callback set 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 10 monclient: shutdown 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:42.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 1 -- 172.21.15.71:0/292342237 >> v1:172.21.15.81:6789/0 conn(0x7f42480a9d70 legacy=0x7f424814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 1 -- 172.21.15.71:0/292342237 shutdown_connections 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 1 -- 172.21.15.71:0/292342237 >> 172.21.15.71:0/292342237 conn(0x7f4248066a00 msgr2=0x7f4248066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 1 -- 172.21.15.71:0/292342237 shutdown_connections 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.055+0000 7f4250110640 1 -- 172.21.15.71:0/292342237 wait complete. 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.056+0000 7f4250110640 10 monclient: build_initial_monmap 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.056+0000 7f4250110640 10 monclient: monmap: 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:42.062 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 1 librados: starting msgr at 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 1 librados: starting objecter 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 1 Processor -- start 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 1 -- start start 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 1 librados: setting wanted keys 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 1 librados: calling monclient init 2024-03-29T11:37:42.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.057+0000 7f4250110640 10 monclient: init 2024-03-29T11:37:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f4250110640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f4250110640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f4250110640 10 monclient(hunting): picked mon.noname-a con 0x7f42480a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f4250110640 10 monclient(hunting): picked mon.noname-c con 0x7f424813f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:42.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f4250110640 10 monclient(hunting): picked mon.noname-b con 0x7f42481457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f424d6ab640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f424813f650 0x7f4248136b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:60058/0 (socket says 172.21.15.71:60058) 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.058+0000 7f424d6ab640 1 -- 172.21.15.71:0/1130955771 learned_addr learned my addr 172.21.15.71:0/1130955771 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 10 client.?.objecter ms_handle_connect 0x7f424813f650 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4248137990 con 0x7f42480a9d70 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42480ac1d0 con 0x7f42481457f0 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42480ac4b0 con 0x7f424813f650 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f4250110640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f4250110640 10 monclient(hunting): authenticate will time out at 1116.214555s 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392762877 0 0) 0x7f42480ac4b0 con 0x7f424813f650 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:42.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.059+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f422c004eb0 con 0x7f424813f650 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071671257 0 0) 0x7f422c004eb0 con 0x7f424813f650 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42480ac4b0 con 0x7f424813f650 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4238001960 con 0x7f424813f650 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 10 monclient(hunting): dump: 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.072 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 10 client.?.objecter ms_handle_connect 0x7f42481457f0 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.060+0000 7f423e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3476826204 0 0) 0x7f42480ac4b0 con 0x7f424813f650 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 >> v1:172.21.15.81:6789/0 conn(0x7f42481457f0 legacy=0x7f4248137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 >> v1:172.21.15.71:6789/0 conn(0x7f42480a9d70 legacy=0x7f42481424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 monclient: found mon.c 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248141f50 con 0x7f424813f650 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 10 monclient: _renew_subs 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42480ab600 con 0x7f424813f650 2024-03-29T11:37:42.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4238001c80 con 0x7f424813f650 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4238003a40 con 0x7f424813f650 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423e7fc640 10 monclient: dump: 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:42.075 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f423ffff640 4 set_mon_vals no callback set 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 10 monclient: _renew_subs 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f424813ff80 con 0x7f424813f650 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.061+0000 7f4250110640 1 librados: init done 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.063+0000 7f4250110640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.063+0000 7f4250110640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.063+0000 7f4250110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.063+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4248142140 con 0x7f424813f650 2024-03-29T11:37:42.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.064+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f4238003ba0 con 0x7f424813f650 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f4238084940 con 0x7f424813f650 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f423e7fc640 10 client.4463.objecter ms_dispatch 0x7f4248000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f423e7fc640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f423e7fc640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f423e7fc640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f424deac640 10 monclient: get_auth_request con 0x7f422c06dc50 auth_method 0 2024-03-29T11:37:42.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.065+0000 7f423e7fc640 10 client.4463.objecter ms_handle_connect 0x7f422c06dc50 2024-03-29T11:37:42.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.072+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4238050220 con 0x7f424813f650 2024-03-29T11:37:42.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.072+0000 7f423e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:42.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.072+0000 7f423e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:42.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.375+0000 7f4250110640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:42.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.375+0000 7f4250110640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:42.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.375+0000 7f4250110640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:42.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.375+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f424813ff80 con 0x7f424813f650 2024-03-29T11:37:42.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.376+0000 7f423e7fc640 1 -- 172.21.15.71:0/1130955771 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f424813ff80 con 0x7f424813f650 2024-03-29T11:37:42.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.376+0000 7f423e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:42.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.376+0000 7f423e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:42.377 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:42.378 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:42.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.378+0000 7f4250110640 10 librados: watch_flush enter 2024-03-29T11:37:42.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 10 librados: watch_flush exit 2024-03-29T11:37:42.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:37:42.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:37:42.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 >> v1:172.21.15.71:6816/1596514235 conn(0x7f422c06dc50 legacy=0x7f422c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 10 monclient: shutdown 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 >> v1:172.21.15.71:6790/0 conn(0x7f424813f650 legacy=0x7f4248136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 shutdown_connections 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 >> 172.21.15.71:0/1130955771 conn(0x7f4248066a00 msgr2=0x7f42480b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 shutdown_connections 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 -- 172.21.15.71:0/1130955771 wait complete. 2024-03-29T11:37:42.383 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:42.379+0000 7f4250110640 1 librados: shutdown 2024-03-29T11:37:42.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:49.203 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.351+0000 7f8bd39c7640 10 monclient: get_monmap_and_config 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.351+0000 7f8bd39c7640 10 monclient: build_initial_monmap 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.351+0000 7f8bd39c7640 10 monclient: monmap: 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:49.351999+0000 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:49.351999+0000 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:49.351 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:49.352 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.351+0000 7f8bd39c7640 1 Processor -- start 2024-03-29T11:37:49.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.352+0000 7f8bd39c7640 1 -- start start 2024-03-29T11:37:49.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.352+0000 7f8bd39c7640 10 monclient: init 2024-03-29T11:37:49.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.353+0000 7f8bd39c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:49.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.353+0000 7f8bd39c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:49.353 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.353+0000 7f8bd39c7640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc1422b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:49.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.353+0000 7f8bd39c7640 10 monclient(hunting): picked mon.noname-a con 0x7f8bcc142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:49.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.354+0000 7f8bd39c7640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc0b17d0 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:49.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.354+0000 7f8bd39c7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc142c80 con 0x7f8bcc142690 2024-03-29T11:37:49.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.354+0000 7f8bd39c7640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0aae40 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.354+0000 7f8bd39c7640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc13fbd0 con 0x7f8bcc1422b0 2024-03-29T11:37:49.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.354+0000 7f8bd39c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:49.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.354+0000 7f8bd39c7640 10 monclient(hunting): authenticate will time out at 1123.509019s 2024-03-29T11:37:49.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.355+0000 7f8bd1f64640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8bcc0b17d0 0x7f8bcc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:59198/0 (socket says 172.21.15.71:59198) 2024-03-29T11:37:49.356 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.355+0000 7f8bd1f64640 1 -- 172.21.15.71:0/977783105 learned_addr learned my addr 172.21.15.71:0/977783105 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.356+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493802834 0 0) 0x7f8bcc0aae40 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.356+0000 7f8bc3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.356+0000 7f8bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.356+0000 7f8bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.356+0000 7f8bc3fff640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.356+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb8003520 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3482978349 0 0) 0x7f8bb8003520 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.357 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 >> v1:172.21.15.71:6790/0 conn(0x7f8bcc1422b0 legacy=0x7f8bcc13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:49.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 >> v1:172.21.15.71:6789/0 conn(0x7f8bcc142690 legacy=0x7f8bcc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:49.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 1 monclient: found mon.noname-c 2024-03-29T11:37:49.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc151940 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8bc4001a10 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.357+0000 7f8bc3fff640 10 monclient: dump: 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.361 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bd39c7640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bd39c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001b70 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bc3fff640 1 -- 172.21.15.71:0/977783105 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8bc4001f80 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bc3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:49.362 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.358+0000 7f8bc3fff640 10 monclient: dump: 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.359+0000 7f8bd39c7640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.359+0000 7f8bd39c7640 4 set_mon_vals no callback set 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.359+0000 7f8bd39c7640 10 monclient: shutdown 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.360+0000 7f8bd39c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:49.363 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.360+0000 7f8bd39c7640 1 -- 172.21.15.71:0/977783105 >> v1:172.21.15.81:6789/0 conn(0x7f8bcc0b17d0 legacy=0x7f8bcc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.360+0000 7f8bd39c7640 1 -- 172.21.15.71:0/977783105 shutdown_connections 2024-03-29T11:37:49.364 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.360+0000 7f8bd39c7640 1 -- 172.21.15.71:0/977783105 >> 172.21.15.71:0/977783105 conn(0x7f8bcc066a00 msgr2=0x7f8bcc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.361+0000 7f8bd39c7640 1 -- 172.21.15.71:0/977783105 shutdown_connections 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.361+0000 7f8bd39c7640 1 -- 172.21.15.71:0/977783105 wait complete. 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.362+0000 7f8bd39c7640 10 monclient: build_initial_monmap 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.362+0000 7f8bd39c7640 10 monclient: monmap: 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:49.365 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 1 librados: starting msgr at 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 1 librados: starting objecter 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 1 Processor -- start 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 1 -- start start 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 1 librados: setting wanted keys 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 1 librados: calling monclient init 2024-03-29T11:37:49.366 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.363+0000 7f8bd39c7640 10 monclient: init 2024-03-29T11:37:49.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.364+0000 7f8bd39c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:49.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.364+0000 7f8bd39c7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:49.367 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 10 monclient(hunting): picked mon.noname-a con 0x7f8bcc0b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 10 monclient(hunting): picked mon.noname-c con 0x7f8bcc1422b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 10 monclient(hunting): picked mon.noname-b con 0x7f8bcc142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0abab0 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc12f680 con 0x7f8bcc142690 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bcc0aca60 con 0x7f8bcc1422b0 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.366+0000 7f8bd39c7640 10 monclient(hunting): authenticate will time out at 1123.520189s 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.367+0000 7f8bd1763640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f8bcc0b17d0 0x7f8bcc0ad720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48207/0 (socket says 172.21.15.71:48207) 2024-03-29T11:37:49.368 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.367+0000 7f8bd1763640 1 -- 172.21.15.71:0/2721969497 learned_addr learned my addr 172.21.15.71:0/2721969497 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:49.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.367+0000 7f8bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f8bcc0b17d0 2024-03-29T11:37:49.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.367+0000 7f8bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:49.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 client.?.objecter ms_handle_connect 0x7f8bcc142690 2024-03-29T11:37:49.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:49.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419843077 0 0) 0x7f8bcc0abab0 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.369 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0004eb0 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226596701 0 0) 0x7f8bcc12f680 con 0x7f8bcc142690 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bcc0abab0 con 0x7f8bcc142690 2024-03-29T11:37:49.370 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486282738 0 0) 0x7f8bb0004eb0 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bcc12f680 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.368+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8bbc004080 con 0x7f8bcc0b17d0 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.371 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 549918912 0 0) 0x7f8bcc0abab0 con 0x7f8bcc142690 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bb0004eb0 con 0x7f8bcc142690 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8bc4001e80 con 0x7f8bcc142690 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.369+0000 7f8bc1ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:49.372 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:49.373 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:49.373 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2928338166 0 0) 0x7f8bb0004eb0 con 0x7f8bcc142690 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 >> v1:172.21.15.71:6790/0 conn(0x7f8bcc1422b0 legacy=0x7f8bcc12e860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 >> v1:172.21.15.71:6789/0 conn(0x7f8bcc0b17d0 legacy=0x7f8bcc0ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 1 monclient: found mon.b 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bcc0acd40 con 0x7f8bcc142690 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:49.376 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bd39c7640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bd39c7640 10 monclient: _renew_subs 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bd39c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.370+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bcc0ad020 con 0x7f8bcc142690 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.371+0000 7f8bd39c7640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.371+0000 7f8bd39c7640 10 monclient: _renew_subs 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.371+0000 7f8bd39c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.371+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bcc0ab540 con 0x7f8bcc142690 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.371+0000 7f8bd39c7640 1 librados: init done 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.372+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bc4001fe0 con 0x7f8bcc142690 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.372+0000 7f8bc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.372+0000 7f8bc37fe640 4 set_mon_vals no callback set 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.372+0000 7f8bd39c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8bc4004e10 con 0x7f8bcc142690 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.377 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bc1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bc1ffb640 10 monclient: dump: 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bd39c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bd39c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.373+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bcc075f10 con 0x7f8bcc142690 2024-03-29T11:37:49.378 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.374+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8bc401ae60 con 0x7f8bcc142690 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f8bc4084e20 con 0x7f8bcc142690 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bc1ffb640 10 client.4441.objecter ms_dispatch 0x7f8bcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bc1ffb640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bc1ffb640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bc1ffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bd1763640 10 monclient: get_auth_request con 0x7f8bb006ddf0 auth_method 0 2024-03-29T11:37:49.379 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.375+0000 7f8bc1ffb640 10 client.4441.objecter ms_handle_connect 0x7f8bb006ddf0 2024-03-29T11:37:49.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.382+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bc4050700 con 0x7f8bcc142690 2024-03-29T11:37:49.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.382+0000 7f8bc1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:49.382 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.382+0000 7f8bc1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:49.714 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:49.714 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:49.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.711+0000 7f8bd39c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.711+0000 7f8bd39c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.711+0000 7f8bd39c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:37:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.711+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bcc0b2270 con 0x7f8bcc142690 2024-03-29T11:37:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.713+0000 7f8bc1ffb640 1 -- 172.21.15.71:0/2721969497 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f8bc4052630 con 0x7f8bcc142690 2024-03-29T11:37:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.713+0000 7f8bc1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:49.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.713+0000 7f8bc1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:49.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.717+0000 7f8bd39c7640 10 librados: watch_flush enter 2024-03-29T11:37:49.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.717+0000 7f8bd39c7640 10 librados: watch_flush exit 2024-03-29T11:37:49.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.717+0000 7f8bd39c7640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T11:37:49.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.717+0000 7f8bd39c7640 10 client.4441.objecter successfully canceled tick 2024-03-29T11:37:49.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 >> v1:172.21.15.71:6816/1596514235 conn(0x7f8bb006ddf0 legacy=0x7f8bb00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 10 monclient: shutdown 2024-03-29T11:37:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 >> v1:172.21.15.81:6789/0 conn(0x7f8bcc142690 legacy=0x7f8bcc12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 shutdown_connections 2024-03-29T11:37:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 >> 172.21.15.71:0/2721969497 conn(0x7f8bcc066a00 msgr2=0x7f8bcc13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:49.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 shutdown_connections 2024-03-29T11:37:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 -- 172.21.15.71:0/2721969497 wait complete. 2024-03-29T11:37:49.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:49.719+0000 7f8bd39c7640 1 librados: shutdown 2024-03-29T11:37:49.735 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:56.541 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:56.694 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.693+0000 7f7fe52ba640 10 monclient: get_monmap_and_config 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.693+0000 7f7fe52ba640 10 monclient: build_initial_monmap 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.693+0000 7f7fe52ba640 10 monclient: monmap: 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:37:56.694455+0000 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:37:56.694455+0000 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:56.695 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.694+0000 7f7fe52ba640 1 Processor -- start 2024-03-29T11:37:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.694+0000 7f7fe52ba640 1 -- start start 2024-03-29T11:37:56.696 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.694+0000 7f7fe52ba640 10 monclient: init 2024-03-29T11:37:56.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:56.697 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient(hunting): picked mon.noname-b con 0x7f7fe0071f40 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe013f060 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe0145ba0 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00aad90 con 0x7f7fe0145ba0 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00aa260 con 0x7f7fe013f060 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00ab000 con 0x7f7fe0071f40 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:56.698 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.695+0000 7f7fe52ba640 10 monclient(hunting): authenticate will time out at 1130.849766s 2024-03-29T11:37:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.698+0000 7f7fdf7fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f7fe0145ba0 0x7f7fe014d830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:39433/0 (socket says 172.21.15.71:39433) 2024-03-29T11:37:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.698+0000 7f7fdf7fe640 1 -- 172.21.15.71:0/2994563485 learned_addr learned my addr 172.21.15.71:0/2994563485 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:56.699 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.699+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819820007 0 0) 0x7f7fe00aad90 con 0x7f7fe0145ba0 2024-03-29T11:37:56.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.699+0000 7f7fddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:56.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.699+0000 7f7fddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:56.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.699+0000 7f7fddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:56.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.699+0000 7f7fddffb640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:37:56.700 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.699+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc8003520 con 0x7f7fe0145ba0 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2268360850 0 0) 0x7f7fc8003520 con 0x7f7fe0145ba0 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 >> v1:172.21.15.71:6790/0 conn(0x7f7fe0071f40 legacy=0x7f7fe00a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 >> v1:172.21.15.81:6789/0 conn(0x7f7fe013f060 legacy=0x7f7fe0145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 1 monclient: found mon.noname-a 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:37:56.703 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00a9d40 con 0x7f7fe0145ba0 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.701+0000 7f7fddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7fd4001690 con 0x7f7fe0145ba0 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: dump: 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:56.704 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fd40018b0 con 0x7f7fe0145ba0 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 1 -- 172.21.15.71:0/2994563485 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7fd4001c40 con 0x7f7fe0145ba0 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fe52ba640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fe52ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fe52ba640 4 set_mon_vals no callback set 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.702+0000 7f7fddffb640 10 monclient: dump: 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:56.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:56.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.703+0000 7f7fe52ba640 10 monclient: shutdown 2024-03-29T11:37:56.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.703+0000 7f7fe52ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:56.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.703+0000 7f7fe52ba640 1 -- 172.21.15.71:0/2994563485 >> v1:172.21.15.71:6789/0 conn(0x7f7fe0145ba0 legacy=0x7f7fe014d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:56.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.707+0000 7f7fe52ba640 1 -- 172.21.15.71:0/2994563485 shutdown_connections 2024-03-29T11:37:56.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.707+0000 7f7fe52ba640 1 -- 172.21.15.71:0/2994563485 >> 172.21.15.71:0/2994563485 conn(0x7f7fe00b3a00 msgr2=0x7f7fe00b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:56.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.709+0000 7f7fe52ba640 1 -- 172.21.15.71:0/2994563485 shutdown_connections 2024-03-29T11:37:56.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.710+0000 7f7fe52ba640 1 -- 172.21.15.71:0/2994563485 wait complete. 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.711+0000 7f7fe52ba640 10 monclient: build_initial_monmap 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.711+0000 7f7fe52ba640 10 monclient: monmap: 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:56.712 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.711+0000 7f7fe52ba640 1 librados: starting msgr at 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.711+0000 7f7fe52ba640 1 librados: starting objecter 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 Processor -- start 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 -- start start 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 librados: setting wanted keys 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 librados: calling monclient init 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient: init 2024-03-29T11:37:56.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient(hunting): picked mon.noname-c con 0x7f7fe0071f40 addr v1:172.21.15.71:6790/0 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient(hunting): picked mon.noname-b con 0x7f7fe013f060 addr v1:172.21.15.81:6789/0 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient(hunting): picked mon.noname-a con 0x7f7fe0145ba0 addr v1:172.21.15.71:6789/0 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0141a50 con 0x7f7fe0145ba0 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe01427a0 con 0x7f7fe013f060 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe0141530 con 0x7f7fe0071f40 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:56.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.712+0000 7f7fe52ba640 10 monclient(hunting): authenticate will time out at 1130.866994s 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.715+0000 7f7fdeffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f7fe0071f40 0x7f7fe01375c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57245/0 (socket says 172.21.15.71:57245) 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.715+0000 7f7fdeffd640 1 -- 172.21.15.71:0/3495393205 learned_addr learned my addr 172.21.15.71:0/3495393205 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 10 client.?.objecter ms_handle_connect 0x7f7fe0071f40 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484076693 0 0) 0x7f7fe0141530 con 0x7f7fe0071f40 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:37:56.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fcc004eb0 con 0x7f7fe0071f40 2024-03-29T11:37:56.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3307819791 0 0) 0x7f7fcc004eb0 con 0x7f7fe0071f40 2024-03-29T11:37:56.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe0141530 con 0x7f7fe0071f40 2024-03-29T11:37:56.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.716+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7fd0001a10 con 0x7f7fe0071f40 2024-03-29T11:37:56.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:56.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 10 monclient(hunting): dump: 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1499541426 0 0) 0x7f7fe0141530 con 0x7f7fe0071f40 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 >> v1:172.21.15.81:6789/0 conn(0x7f7fe013f060 legacy=0x7f7fe01d6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:56.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.717+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 >> v1:172.21.15.71:6789/0 conn(0x7f7fe0145ba0 legacy=0x7f7fe01e27d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:56.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:56.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 1 monclient: found mon.c 2024-03-29T11:37:56.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:56.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:56.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0141190 con 0x7f7fe0071f40 2024-03-29T11:37:56.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:56.721 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fe52ba640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fe52ba640 10 monclient: _renew_subs 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fe52ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.719+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe0138260 con 0x7f7fe0071f40 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.720+0000 7f7fe52ba640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.720+0000 7f7fe52ba640 10 monclient: _renew_subs 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.720+0000 7f7fe52ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.720+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe0143080 con 0x7f7fe0071f40 2024-03-29T11:37:56.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.720+0000 7f7fe52ba640 1 librados: init done 2024-03-29T11:37:56.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.721+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fd0001c70 con 0x7f7fe0071f40 2024-03-29T11:37:56.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.721+0000 7f7fc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:56.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.721+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7fd0003b20 con 0x7f7fe0071f40 2024-03-29T11:37:56.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.721+0000 7f7fc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:56.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.722+0000 7f7fc77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:56.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.722+0000 7f7fc77fe640 10 monclient: dump: 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:37:56.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.722+0000 7f7fdd7fa640 4 set_mon_vals no callback set 2024-03-29T11:37:56.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.724+0000 7f7fe52ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:56.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.724+0000 7f7fe52ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:56.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.724+0000 7f7fe52ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:56.725 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.724+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc0005080 con 0x7f7fe0071f40 2024-03-29T11:37:56.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.725+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f7fd0003c80 con 0x7f7fe0071f40 2024-03-29T11:37:56.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.726+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f7fd0084c90 con 0x7f7fe0071f40 2024-03-29T11:37:56.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.726+0000 7f7fc77fe640 10 client.4469.objecter ms_dispatch 0x7f7fe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:56.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.726+0000 7f7fc77fe640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:56.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.726+0000 7f7fc77fe640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:56.726 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.726+0000 7f7fc77fe640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-29T11:37:56.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.727+0000 7f7fde7fc640 10 monclient: get_auth_request con 0x7f7fcc06dbc0 auth_method 0 2024-03-29T11:37:56.727 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.727+0000 7f7fc77fe640 10 client.4469.objecter ms_handle_connect 0x7f7fcc06dbc0 2024-03-29T11:37:56.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.734+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fd00503a0 con 0x7f7fe0071f40 2024-03-29T11:37:56.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.734+0000 7f7fc77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:56.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:56.734+0000 7f7fc77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:57.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.059+0000 7f7fe52ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:57.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.059+0000 7f7fe52ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:57.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.059+0000 7f7fe52ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:37:57.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.059+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fc0005330 con 0x7f7fe0071f40 2024-03-29T11:37:57.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.060+0000 7f7fc77fe640 1 -- 172.21.15.71:0/3495393205 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f7fd00524a0 con 0x7f7fe0071f40 2024-03-29T11:37:57.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.060+0000 7f7fc77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:57.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.060+0000 7f7fc77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:57.061 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:37:57.061 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:57.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.064+0000 7f7fe52ba640 10 librados: watch_flush enter 2024-03-29T11:37:57.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.064+0000 7f7fe52ba640 10 librados: watch_flush exit 2024-03-29T11:37:57.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.064+0000 7f7fe52ba640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-29T11:37:57.064 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.064+0000 7f7fe52ba640 10 client.4469.objecter successfully canceled tick 2024-03-29T11:37:57.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.065+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 >> v1:172.21.15.71:6816/1596514235 conn(0x7f7fcc06dbc0 legacy=0x7f7fcc070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.065+0000 7f7fe52ba640 10 monclient: shutdown 2024-03-29T11:37:57.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.065+0000 7f7fe52ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:57.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.065+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 >> v1:172.21.15.71:6790/0 conn(0x7f7fe0071f40 legacy=0x7f7fe01375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.065+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 shutdown_connections 2024-03-29T11:37:57.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.065+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 >> 172.21.15.71:0/3495393205 conn(0x7f7fe00b3a00 msgr2=0x7f7fe00b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:57.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.066+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 shutdown_connections 2024-03-29T11:37:57.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.067+0000 7f7fe52ba640 1 -- 172.21.15.71:0/3495393205 wait complete. 2024-03-29T11:37:57.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:37:57.067+0000 7f7fe52ba640 1 librados: shutdown 2024-03-29T11:37:57.087 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:03.895 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.045+0000 7efe6ee49640 10 monclient: get_monmap_and_config 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.045+0000 7efe6ee49640 10 monclient: build_initial_monmap 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.045+0000 7efe6ee49640 10 monclient: monmap: 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:04.046656+0000 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:04.046656+0000 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:04.046 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:04.047 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.047+0000 7efe6ee49640 1 Processor -- start 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.049+0000 7efe6ee49640 1 -- start start 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.049+0000 7efe6ee49640 10 monclient: init 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient(hunting): picked mon.noname-b con 0x7efe681457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient(hunting): picked mon.noname-c con 0x7efe6813f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient(hunting): picked mon.noname-a con 0x7efe680a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe680aaef0 con 0x7efe680a9d70 2024-03-29T11:38:04.051 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe68145bd0 con 0x7efe6813f650 2024-03-29T11:38:04.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe68145eb0 con 0x7efe681457f0 2024-03-29T11:38:04.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:04.052 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.050+0000 7efe6ee49640 10 monclient(hunting): authenticate will time out at 1138.204980s 2024-03-29T11:38:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.055+0000 7efe6d3e6640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efe680a9d70 0x7efe6814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49821/0 (socket says 172.21.15.71:49821) 2024-03-29T11:38:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.055+0000 7efe6d3e6640 1 -- 172.21.15.71:0/3016466791 learned_addr learned my addr 172.21.15.71:0/3016466791 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.055+0000 7efe6d3e6640 1 -- 172.21.15.71:0/3016466791 >> v1:172.21.15.71:6789/0 conn(0x7efe680a9d70 legacy=0x7efe6814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T11:38:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.055+0000 7efe6d3e6640 1 -- 172.21.15.71:0/3016466791 >> v1:172.21.15.71:6789/0 conn(0x7efe680a9d70 legacy=0x7efe6814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:38:04.056 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.055+0000 7efe6d3e6640 1 --1- 172.21.15.71:0/3016466791 >> v1:172.21.15.71:6789/0 conn(0x7efe680a9d70 0x7efe6814d8c0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:38:04.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327125682 0 0) 0x7efe68145eb0 con 0x7efe681457f0 2024-03-29T11:38:04.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:04.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe58003520 con 0x7efe681457f0 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.058+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193942574 0 0) 0x7efe68145bd0 con 0x7efe6813f650 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe68145eb0 con 0x7efe6813f650 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 885023259 0 0) 0x7efe58003520 con 0x7efe681457f0 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 >> v1:172.21.15.81:6789/0 conn(0x7efe6813f650 legacy=0x7efe680b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 >> v1:172.21.15.71:6789/0 conn(0x7efe680a9d70 legacy=0x7efe6814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:04.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 monclient: found mon.noname-b 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68146340 con 0x7efe681457f0 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efe5c001690 con 0x7efe681457f0 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.059+0000 7efe677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.060+0000 7efe677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.060+0000 7efe677fe640 10 monclient: dump: 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.062 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.063 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.063 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:04.063 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:04.063 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:04.063 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:04.063 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.063+0000 7efe6ee49640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.063+0000 7efe6ee49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe5c0017f0 con 0x7efe681457f0 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe677fe640 1 -- 172.21.15.71:0/3016466791 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efe5c001c00 con 0x7efe681457f0 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe6ee49640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe6ee49640 4 set_mon_vals no callback set 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:04.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.064+0000 7efe677fe640 10 monclient: dump: 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:04.066 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:04.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.065+0000 7efe6ee49640 10 monclient: shutdown 2024-03-29T11:38:04.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.065+0000 7efe6ee49640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:04.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.065+0000 7efe6ee49640 1 -- 172.21.15.71:0/3016466791 >> v1:172.21.15.71:6790/0 conn(0x7efe681457f0 legacy=0x7efe6813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.065+0000 7efe6ee49640 1 -- 172.21.15.71:0/3016466791 shutdown_connections 2024-03-29T11:38:04.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.065+0000 7efe6ee49640 1 -- 172.21.15.71:0/3016466791 >> 172.21.15.71:0/3016466791 conn(0x7efe68066a00 msgr2=0x7efe68066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:04.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.067+0000 7efe6ee49640 1 -- 172.21.15.71:0/3016466791 shutdown_connections 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.067+0000 7efe6ee49640 1 -- 172.21.15.71:0/3016466791 wait complete. 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.068+0000 7efe6ee49640 10 monclient: build_initial_monmap 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.068+0000 7efe6ee49640 10 monclient: monmap: 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:04.070 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 1 librados: starting msgr at 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 1 librados: starting objecter 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 1 Processor -- start 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 1 -- start start 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 1 librados: setting wanted keys 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 1 librados: calling monclient init 2024-03-29T11:38:04.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.069+0000 7efe6ee49640 10 monclient: init 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient(hunting): picked mon.noname-b con 0x7efe680a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient(hunting): picked mon.noname-a con 0x7efe6813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient(hunting): picked mon.noname-c con 0x7efe681457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6813a790 con 0x7efe6813f650 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe681d7210 con 0x7efe680a9d70 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6813fe10 con 0x7efe681457f0 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:04.073 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.070+0000 7efe6ee49640 10 monclient(hunting): authenticate will time out at 1138.224809s 2024-03-29T11:38:04.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe6cbe5640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efe680a9d70 0x7efe681e76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:56697/0 (socket says 172.21.15.71:56697) 2024-03-29T11:38:04.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe6d3e6640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7efe681457f0 0x7efe681d6b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55569/0 (socket says 172.21.15.71:55569) 2024-03-29T11:38:04.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe6d3e6640 1 -- 172.21.15.71:0/2657203908 learned_addr learned my addr 172.21.15.71:0/2657203908 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 client.?.objecter ms_handle_connect 0x7efe681457f0 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724866452 0 0) 0x7efe6813fe10 con 0x7efe681457f0 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:38:04.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c004eb0 con 0x7efe681457f0 2024-03-29T11:38:04.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 664817132 0 0) 0x7efe4c004eb0 con 0x7efe681457f0 2024-03-29T11:38:04.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe4c005300 con 0x7efe681457f0 2024-03-29T11:38:04.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efe60001890 con 0x7efe681457f0 2024-03-29T11:38:04.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): dump: 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:04.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.078+0000 7efe657fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3417533369 0 0) 0x7efe4c005300 con 0x7efe681457f0 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 >> v1:172.21.15.81:6789/0 conn(0x7efe680a9d70 legacy=0x7efe681e76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 >> v1:172.21.15.71:6789/0 conn(0x7efe6813f650 legacy=0x7efe681d63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 1 monclient: found mon.c 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:04.086 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe68139600 con 0x7efe681457f0 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe60001e40 con 0x7efe681457f0 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.079+0000 7efe667fc640 4 set_mon_vals no callback set 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efe60001890 con 0x7efe681457f0 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe657fa640 10 monclient: dump: 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:04.087 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 10 monclient: _renew_subs 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe68139850 con 0x7efe681457f0 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 10 monclient: _renew_subs 2024-03-29T11:38:04.088 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe681da430 con 0x7efe681457f0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.080+0000 7efe6ee49640 1 librados: init done 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.082+0000 7efe6ee49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.082+0000 7efe6ee49640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.082+0000 7efe6ee49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6813ac00 con 0x7efe681457f0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7efe60001fb0 con 0x7efe681457f0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7efe600847b0 con 0x7efe681457f0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe657fa640 10 client.4475.objecter ms_dispatch 0x7efe68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe657fa640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe657fa640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe657fa640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.083+0000 7efe6cbe5640 10 monclient: get_auth_request con 0x7efe4c06e2a0 auth_method 0 2024-03-29T11:38:04.089 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.084+0000 7efe657fa640 10 client.4475.objecter ms_handle_connect 0x7efe4c06e2a0 2024-03-29T11:38:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.092+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe6813ac00 con 0x7efe681457f0 2024-03-29T11:38:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.092+0000 7efe657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:04.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.092+0000 7efe657fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:04.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.422+0000 7efe6ee49640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:04.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.423+0000 7efe6ee49640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:04.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.423+0000 7efe6ee49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:04.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.423+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe680b7530 con 0x7efe681457f0 2024-03-29T11:38:04.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.423+0000 7efe657fa640 1 -- 172.21.15.71:0/2657203908 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7efe6813ac00 con 0x7efe681457f0 2024-03-29T11:38:04.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.423+0000 7efe657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:04.425 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.423+0000 7efe657fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:04.426 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:04.426 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:04.428 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.427+0000 7efe6ee49640 10 librados: watch_flush enter 2024-03-29T11:38:04.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 10 librados: watch_flush exit 2024-03-29T11:38:04.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:38:04.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:38:04.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 >> v1:172.21.15.71:6816/1596514235 conn(0x7efe4c06e2a0 legacy=0x7efe4c070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.432 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 10 monclient: shutdown 2024-03-29T11:38:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.428+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 >> v1:172.21.15.71:6790/0 conn(0x7efe681457f0 legacy=0x7efe681d6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.433 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.429+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 shutdown_connections 2024-03-29T11:38:04.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.429+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 >> 172.21.15.71:0/2657203908 conn(0x7efe68066a00 msgr2=0x7efe681dabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:04.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.430+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 shutdown_connections 2024-03-29T11:38:04.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.430+0000 7efe6ee49640 1 -- 172.21.15.71:0/2657203908 wait complete. 2024-03-29T11:38:04.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:04.430+0000 7efe6ee49640 1 librados: shutdown 2024-03-29T11:38:04.448 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:11.255 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:11.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.400+0000 7fcec19f4640 10 monclient: get_monmap_and_config 2024-03-29T11:38:11.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.400+0000 7fcec19f4640 10 monclient: build_initial_monmap 2024-03-29T11:38:11.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.400+0000 7fcec19f4640 10 monclient: monmap: 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:11.401330+0000 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:11.401330+0000 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:11.401 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:11.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.401+0000 7fcec19f4640 1 Processor -- start 2024-03-29T11:38:11.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.401+0000 7fcec19f4640 1 -- start start 2024-03-29T11:38:11.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.401+0000 7fcec19f4640 10 monclient: init 2024-03-29T11:38:11.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:11.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:11.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient(hunting): picked mon.noname-b con 0x7fcebc1457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:11.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fceba7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcebc13f650 0x7fcebc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46818/0 (socket says 172.21.15.71:46818) 2024-03-29T11:38:11.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fceba7fc640 1 -- 172.21.15.71:0/1898432160 learned_addr learned my addr 172.21.15.71:0/1898432160 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:11.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient(hunting): picked mon.noname-a con 0x7fcebc13f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:11.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient(hunting): picked mon.noname-c con 0x7fcebc0a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc0aaef0 con 0x7fcebc13f650 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc145bd0 con 0x7fcebc0a9d70 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc145eb0 con 0x7fcebc1457f0 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fcec19f4640 10 monclient(hunting): authenticate will time out at 1145.556607s 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952905493 0 0) 0x7fcebc145eb0 con 0x7fcebc1457f0 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.402+0000 7fceb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce9c003520 con 0x7fcebc1457f0 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72644321 0 0) 0x7fcebc0aaef0 con 0x7fcebc13f650 2024-03-29T11:38:11.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc145eb0 con 0x7fcebc13f650 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748781398 0 0) 0x7fce9c003520 con 0x7fcebc1457f0 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 >> v1:172.21.15.81:6789/0 conn(0x7fcebc0a9d70 legacy=0x7fcebc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 >> v1:172.21.15.71:6789/0 conn(0x7fcebc13f650 legacy=0x7fcebc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc146340 con 0x7fcebc1457f0 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:11.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fcec19f4640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fcec19f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fcea4001690 con 0x7fcebc1457f0 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fceb9ffb640 10 monclient: dump: 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:11.408 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.403+0000 7fcec19f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.404+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcea4001870 con 0x7fcebc1457f0 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.404+0000 7fceb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.404+0000 7fcec19f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.404+0000 7fcec19f4640 4 set_mon_vals no callback set 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 10 monclient: shutdown 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 >> v1:172.21.15.71:6790/0 conn(0x7fcebc1457f0 legacy=0x7fcebc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fceb9ffb640 1 -- 172.21.15.71:0/1898432160 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fcea4001690 con 0x7fcebc1457f0 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fceb9ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 shutdown_connections 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 >> 172.21.15.71:0/1898432160 conn(0x7fcebc066a00 msgr2=0x7fcebc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 shutdown_connections 2024-03-29T11:38:11.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.405+0000 7fcec19f4640 1 -- 172.21.15.71:0/1898432160 wait complete. 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.406+0000 7fcec19f4640 10 monclient: build_initial_monmap 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.406+0000 7fcec19f4640 10 monclient: monmap: 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.406+0000 7fcec19f4640 1 librados: starting msgr at 2024-03-29T11:38:11.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.406+0000 7fcec19f4640 1 librados: starting objecter 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.406+0000 7fcec19f4640 1 Processor -- start 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.407+0000 7fcec19f4640 1 -- start start 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.407+0000 7fcec19f4640 1 librados: setting wanted keys 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.407+0000 7fcec19f4640 1 librados: calling monclient init 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.407+0000 7fcec19f4640 10 monclient: init 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.407+0000 7fcec19f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.407+0000 7fcec19f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 10 monclient(hunting): picked mon.noname-c con 0x7fcebc0a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceba7fc640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7fcebc13f650 0x7fcebc1395c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48076/0 (socket says 172.21.15.71:48076) 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcebaffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fcebc0a9d70 0x7fcebc1db520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51718/0 (socket says 172.21.15.71:51718) 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceba7fc640 1 -- 172.21.15.71:0/4093086134 learned_addr learned my addr 172.21.15.71:0/4093086134 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 10 monclient(hunting): picked mon.noname-a con 0x7fcebc13f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 10 monclient(hunting): picked mon.noname-b con 0x7fcebc1457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:11.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 client.?.objecter ms_handle_connect 0x7fcebc0a9d70 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc070250 con 0x7fcebc13f650 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 client.?.objecter ms_handle_connect 0x7fcebc13f650 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc13a3e0 con 0x7fcebc1457f0 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc06f320 con 0x7fcebc0a9d70 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fcec19f4640 10 monclient(hunting): authenticate will time out at 1145.562620s 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193181762 0 0) 0x7fcebc070250 con 0x7fcebc13f650 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea0004eb0 con 0x7fcebc13f650 2024-03-29T11:38:11.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558218480 0 0) 0x7fcebc06f320 con 0x7fcebc0a9d70 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.408+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcebc070250 con 0x7fcebc0a9d70 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 client.?.objecter ms_handle_connect 0x7fcebc1457f0 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4039444401 0 0) 0x7fcebc070250 con 0x7fcebc0a9d70 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcebc06f320 con 0x7fcebc0a9d70 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2816630021 0 0) 0x7fcea0004eb0 con 0x7fcebc13f650 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcebc070250 con 0x7fcebc13f650 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fcea4001c40 con 0x7fcebc0a9d70 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.418 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): dump: 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fceac001930 con 0x7fcebc13f650 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.419 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient(hunting): dump: 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3418073623 0 0) 0x7fcebc06f320 con 0x7fcebc0a9d70 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 >> v1:172.21.15.81:6789/0 conn(0x7fcebc1457f0 legacy=0x7fcebc139cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 >> v1:172.21.15.71:6789/0 conn(0x7fcebc13f650 legacy=0x7fcebc1395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.420 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 monclient: found mon.c 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc13a780 con 0x7fcebc0a9d70 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fceb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 10 monclient: _renew_subs 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc06fe50 con 0x7fcebc0a9d70 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 10 monclient: _renew_subs 2024-03-29T11:38:11.421 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1dbc30 con 0x7fcebc0a9d70 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.409+0000 7fcec19f4640 1 librados: init done 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcea40038a0 con 0x7fcebc0a9d70 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fcea4003c70 con 0x7fcebc0a9d70 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb3fff640 10 monclient: dump: 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.422 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.410+0000 7fceb97fa640 4 set_mon_vals no callback set 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.411+0000 7fcec19f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.411+0000 7fcec19f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.411+0000 7fcec19f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.411+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce94005080 con 0x7fcebc0a9d70 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.412+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fcea4003dd0 con 0x7fcebc0a9d70 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceba7fc640 10 monclient: get_auth_request con 0x7fcea006dfa0 auth_method 0 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fcea4084aa0 con 0x7fcebc0a9d70 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceb3fff640 10 client.4481.objecter ms_dispatch 0x7fcebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceb3fff640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:11.423 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceb3fff640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:11.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceb3fff640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:38:11.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.413+0000 7fceb3fff640 10 client.4481.objecter ms_handle_connect 0x7fcea006dfa0 2024-03-29T11:38:11.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.420+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcea4050380 con 0x7fcebc0a9d70 2024-03-29T11:38:11.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.420+0000 7fceb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:11.424 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.420+0000 7fceb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.773+0000 7fcec19f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.773+0000 7fcec19f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.773+0000 7fcec19f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:11.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.773+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce94005330 con 0x7fcebc0a9d70 2024-03-29T11:38:11.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.774+0000 7fceb3fff640 1 -- 172.21.15.71:0/4093086134 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fcea40522b0 con 0x7fcebc0a9d70 2024-03-29T11:38:11.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.774+0000 7fceb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:11.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.774+0000 7fceb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:11.775 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:11.775 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 10 librados: watch_flush enter 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 10 librados: watch_flush exit 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 >> v1:172.21.15.71:6816/1596514235 conn(0x7fcea006dfa0 legacy=0x7fcea0070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 10 monclient: shutdown 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 >> v1:172.21.15.71:6790/0 conn(0x7fcebc0a9d70 legacy=0x7fcebc1db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 shutdown_connections 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 >> 172.21.15.71:0/4093086134 conn(0x7fcebc066a00 msgr2=0x7fcebc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:11.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 shutdown_connections 2024-03-29T11:38:11.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 -- 172.21.15.71:0/4093086134 wait complete. 2024-03-29T11:38:11.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:11.777+0000 7fcec19f4640 1 librados: shutdown 2024-03-29T11:38:11.791 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:18.599 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:18.752 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.751+0000 7f886b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.751+0000 7f886b59e640 10 monclient: build_initial_monmap 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.751+0000 7f886b59e640 10 monclient: monmap: 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:18.752686+0000 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:18.752686+0000 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:18.753 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:18.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.752+0000 7f886b59e640 1 Processor -- start 2024-03-29T11:38:18.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.752+0000 7f886b59e640 1 -- start start 2024-03-29T11:38:18.754 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.752+0000 7f886b59e640 10 monclient: init 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.753+0000 7f886b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.753+0000 7f886b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8864006e30 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f8864035e00 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f88640c9670 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8864035970 con 0x7f8864035e00 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8864036280 con 0x7f8864006e30 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8864035640 con 0x7f88640c9670 2024-03-29T11:38:18.756 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:18.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.754+0000 7f886b59e640 10 monclient(hunting): authenticate will time out at 1152.908125s 2024-03-29T11:38:18.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886a59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8864006e30 0x7f88640cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:42577/0 (socket says 172.21.15.71:42577) 2024-03-29T11:38:18.757 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886a59c640 1 -- 172.21.15.71:0/1493696009 learned_addr learned my addr 172.21.15.71:0/1493696009 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421804082 0 0) 0x7f8864036280 con 0x7f8864006e30 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886959a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886959a640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.757+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8854003520 con 0x7f8864006e30 2024-03-29T11:38:18.758 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.758+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526365097 0 0) 0x7f8854003520 con 0x7f8864006e30 2024-03-29T11:38:18.759 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.758+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 >> v1:172.21.15.71:6790/0 conn(0x7f88640c9670 legacy=0x7f88640d7560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:18.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 >> v1:172.21.15.71:6789/0 conn(0x7f8864035e00 legacy=0x7f88640c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:18.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:18.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 1 monclient: found mon.noname-c 2024-03-29T11:38:18.761 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:18.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:38:18.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8864034e20 con 0x7f8864006e30 2024-03-29T11:38:18.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: _finish_auth 0 2024-03-29T11:38:18.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:18.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:18.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886b59e640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f886c04f730 con 0x7f8864006e30 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: dump: 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:18.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 1 -- 172.21.15.71:0/1493696009 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f886c0008d0 con 0x7f8864006e30 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.761+0000 7f886b59e640 4 set_mon_vals no callback set 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.762+0000 7f886b59e640 10 monclient: shutdown 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.762+0000 7f886b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.762+0000 7f886b59e640 1 -- 172.21.15.71:0/1493696009 >> v1:172.21.15.81:6789/0 conn(0x7f8864006e30 legacy=0x7f88640cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.762+0000 7f886b59e640 1 -- 172.21.15.71:0/1493696009 shutdown_connections 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.762+0000 7f886b59e640 1 -- 172.21.15.71:0/1493696009 >> 172.21.15.71:0/1493696009 conn(0x7f8864005c00 msgr2=0x7f8864006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:18.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.763+0000 7f886b59e640 1 -- 172.21.15.71:0/1493696009 shutdown_connections 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.764+0000 7f886b59e640 1 -- 172.21.15.71:0/1493696009 wait complete. 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 10 monclient: build_initial_monmap 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 10 monclient: monmap: 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:18.767 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:18.768 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:18.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 1 librados: starting msgr at 2024-03-29T11:38:18.768 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 1 librados: starting objecter 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 1 Processor -- start 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 1 -- start start 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 1 librados: setting wanted keys 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 1 librados: calling monclient init 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.765+0000 7f886b59e640 10 monclient: init 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f8864006e30 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f88640c9670 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f8864030ff0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:18.772 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886402ed80 con 0x7f88640c9670 2024-03-29T11:38:18.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640301e0 con 0x7f8864006e30 2024-03-29T11:38:18.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88640c1b60 con 0x7f8864030ff0 2024-03-29T11:38:18.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:18.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.766+0000 7f886b59e640 10 monclient(hunting): authenticate will time out at 1152.920809s 2024-03-29T11:38:18.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f8869d9b640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f88640c9670 0x7f88640338b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:34517/0 (socket says 172.21.15.71:34517) 2024-03-29T11:38:18.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f8869d9b640 1 -- 172.21.15.71:0/1209904616 learned_addr learned my addr 172.21.15.71:0/1209904616 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 client.?.objecter ms_handle_connect 0x7f88640c9670 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 363567302 0 0) 0x7f886402ed80 con 0x7f88640c9670 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f884c004eb0 con 0x7f88640c9670 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 client.?.objecter ms_handle_connect 0x7f8864006e30 2024-03-29T11:38:18.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893156316 0 0) 0x7f88640301e0 con 0x7f8864006e30 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.776+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886402ed80 con 0x7f8864006e30 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3504097955 0 0) 0x7f884c004eb0 con 0x7f88640c9670 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88640301e0 con 0x7f88640c9670 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f88600018b0 con 0x7f88640c9670 2024-03-29T11:38:18.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 10 monclient(hunting): dump: 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.777+0000 7f885affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1443468784 0 0) 0x7f886402ed80 con 0x7f8864006e30 2024-03-29T11:38:18.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f884c004eb0 con 0x7f8864006e30 2024-03-29T11:38:18.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4102762375 0 0) 0x7f88640301e0 con 0x7f88640c9670 2024-03-29T11:38:18.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 >> v1:172.21.15.71:6790/0 conn(0x7f8864030ff0 legacy=0x7f886402fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:18.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 >> v1:172.21.15.81:6789/0 conn(0x7f8864006e30 legacy=0x7f886402d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 monclient: found mon.a 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.778+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88640c1cf0 con 0x7f88640c9670 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f885affd640 10 monclient: _finish_auth 0 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f885affd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f885affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 10 monclient: _renew_subs 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88640c1f40 con 0x7f88640c9670 2024-03-29T11:38:18.783 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 10 monclient: _renew_subs 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88640c2420 con 0x7f88640c9670 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.779+0000 7f886b59e640 1 librados: init done 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8860001a80 con 0x7f88640c9670 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f885affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f88600038a0 con 0x7f88640c9670 2024-03-29T11:38:18.784 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f885affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:18.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f885affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:18.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f885affd640 10 monclient: dump: 2024-03-29T11:38:18.785 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:18.785 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:18.785 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:18.785 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.780+0000 7f8868d99640 4 set_mon_vals no callback set 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.784+0000 7f886b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:18.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.784+0000 7f886b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:18.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.784+0000 7f886b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:18.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.784+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8850005080 con 0x7f88640c9670 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.785+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8860003a00 con 0x7f88640c9670 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.786+0000 7f886a59c640 10 monclient: get_auth_request con 0x7f884c06de70 auth_method 0 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.786+0000 7f885affd640 10 client.4440.objecter ms_handle_connect 0x7f884c06de70 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.787+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f8860084990 con 0x7f88640c9670 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.787+0000 7f885affd640 10 client.4440.objecter ms_dispatch 0x7f886c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.787+0000 7f885affd640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.787+0000 7f885affd640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:18.791 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.787+0000 7f885affd640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T11:38:18.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.797+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8860050270 con 0x7f88640c9670 2024-03-29T11:38:18.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.797+0000 7f885affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:18.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:18.797+0000 7f885affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:19.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.207+0000 7f886b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:19.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.207+0000 7f886b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:19.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.207+0000 7f886b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:19.208 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.207+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8850005330 con 0x7f88640c9670 2024-03-29T11:38:19.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.213+0000 7f885affd640 1 -- 172.21.15.71:0/1209904616 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f88600521a0 con 0x7f88640c9670 2024-03-29T11:38:19.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.213+0000 7f885affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:19.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.213+0000 7f885affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:19.213 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:19.213 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 10 librados: watch_flush enter 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 10 librados: watch_flush exit 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 10 client.4440.objecter successfully canceled tick 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 >> v1:172.21.15.71:6816/1596514235 conn(0x7f884c06de70 legacy=0x7f884c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 10 monclient: shutdown 2024-03-29T11:38:19.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:19.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.216+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 >> v1:172.21.15.71:6789/0 conn(0x7f88640c9670 legacy=0x7f88640338b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.217+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 shutdown_connections 2024-03-29T11:38:19.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.217+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 >> 172.21.15.71:0/1209904616 conn(0x7f8864005c00 msgr2=0x7f88640c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:19.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.217+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 shutdown_connections 2024-03-29T11:38:19.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.217+0000 7f886b59e640 1 -- 172.21.15.71:0/1209904616 wait complete. 2024-03-29T11:38:19.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:19.217+0000 7f886b59e640 1 librados: shutdown 2024-03-29T11:38:19.234 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:26.041 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.201+0000 7fae93d7b640 10 monclient: get_monmap_and_config 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.201+0000 7fae93d7b640 10 monclient: build_initial_monmap 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.201+0000 7fae93d7b640 10 monclient: monmap: 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:26.202494+0000 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:26.202494+0000 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:26.202 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:26.203 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:26.203 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:26.203 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:26.203 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:26.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.202+0000 7fae93d7b640 1 Processor -- start 2024-03-29T11:38:26.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.203+0000 7fae93d7b640 1 -- start start 2024-03-29T11:38:26.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.203+0000 7fae93d7b640 10 monclient: init 2024-03-29T11:38:26.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:26.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c1457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c13f650 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c0a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0aaef0 con 0x7fae8c0a9d70 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c145bd0 con 0x7fae8c1457f0 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c145eb0 con 0x7fae8c13f650 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:26.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.204+0000 7fae93d7b640 10 monclient(hunting): authenticate will time out at 1160.358208s 2024-03-29T11:38:26.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.205+0000 7fae91316640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fae8c13f650 0x7fae8c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52632/0 (socket says 172.21.15.71:52632) 2024-03-29T11:38:26.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.206+0000 7fae91316640 1 -- 172.21.15.71:0/2666325448 learned_addr learned my addr 172.21.15.71:0/2666325448 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:26.209 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.207+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896533038 0 0) 0x7fae8c145eb0 con 0x7fae8c13f650 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.207+0000 7fae90b15640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.207+0000 7fae90b15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.207+0000 7fae90b15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.207+0000 7fae90b15640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.207+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae78003520 con 0x7fae8c13f650 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3384828362 0 0) 0x7fae78003520 con 0x7fae8c13f650 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 >> v1:172.21.15.81:6789/0 conn(0x7fae8c1457f0 legacy=0x7fae8c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 >> v1:172.21.15.71:6789/0 conn(0x7fae8c0a9d70 legacy=0x7fae8c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: _finish_hunting 0 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 1 monclient: found mon.noname-b 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:38:26.210 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c146340 con 0x7fae8c13f650 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: _finish_auth 0 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: _check_auth_tickets 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fae88001670 con 0x7fae8c13f650 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae90b15640 10 monclient: dump: 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:26.211 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae93d7b640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.208+0000 7fae93d7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae88001840 con 0x7fae8c13f650 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae90b15640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae90b15640 1 -- 172.21.15.71:0/2666325448 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fae88001c90 con 0x7fae8c13f650 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae90b15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae90b15640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae90b15640 10 monclient: dump: 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.212 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae93d7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:26.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.209+0000 7fae93d7b640 4 set_mon_vals no callback set 2024-03-29T11:38:26.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 10 monclient: shutdown 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 1 -- 172.21.15.71:0/2666325448 >> v1:172.21.15.71:6790/0 conn(0x7fae8c13f650 legacy=0x7fae8c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 1 -- 172.21.15.71:0/2666325448 shutdown_connections 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 1 -- 172.21.15.71:0/2666325448 >> 172.21.15.71:0/2666325448 conn(0x7fae8c066a00 msgr2=0x7fae8c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 1 -- 172.21.15.71:0/2666325448 shutdown_connections 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.210+0000 7fae93d7b640 1 -- 172.21.15.71:0/2666325448 wait complete. 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.211+0000 7fae93d7b640 10 monclient: build_initial_monmap 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.211+0000 7fae93d7b640 10 monclient: monmap: 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:26.215 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 1 librados: starting msgr at 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 1 librados: starting objecter 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 1 Processor -- start 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 1 -- start start 2024-03-29T11:38:26.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 1 librados: setting wanted keys 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 1 librados: calling monclient init 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.212+0000 7fae93d7b640 10 monclient: init 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.213+0000 7fae93d7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.213+0000 7fae93d7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.213+0000 7fae93d7b640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c0a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae91b17640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fae8c0a9d70 0x7fae8c13c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52914/0 (socket says 172.21.15.71:52914) 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c13f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c1457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c13e320 con 0x7fae8c13f650 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c130720 con 0x7fae8c1457f0 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c144eb0 con 0x7fae8c0a9d70 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae93d7b640 10 monclient(hunting): authenticate will time out at 1160.368264s 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae91b17640 1 -- 172.21.15.71:0/1458623259 learned_addr learned my addr 172.21.15.71:0/1458623259 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:26.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae827fc640 10 client.?.objecter ms_handle_connect 0x7fae8c0a9d70 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.214+0000 7fae827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.216+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031436033 0 0) 0x7fae8c144eb0 con 0x7fae8c0a9d70 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.216+0000 7fae827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.216+0000 7fae827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.216+0000 7fae827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.216+0000 7fae827fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:38:26.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.216+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae70004eb0 con 0x7fae8c0a9d70 2024-03-29T11:38:26.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 client.?.objecter ms_handle_connect 0x7fae8c13f650 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 client.?.objecter ms_handle_connect 0x7fae8c1457f0 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002851336 0 0) 0x7fae8c13e320 con 0x7fae8c13f650 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae700051d0 con 0x7fae8c13f650 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295797438 0 0) 0x7fae8c130720 con 0x7fae8c1457f0 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.218+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae8c13e320 con 0x7fae8c1457f0 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847036534 0 0) 0x7fae700051d0 con 0x7fae8c13f650 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae700057d0 con 0x7fae8c13f650 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fae88001c90 con 0x7fae8c13f650 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient(hunting): dump: 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.222 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1941657483 0 0) 0x7fae700057d0 con 0x7fae8c13f650 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 >> v1:172.21.15.71:6790/0 conn(0x7fae8c0a9d70 legacy=0x7fae8c13c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 >> v1:172.21.15.81:6789/0 conn(0x7fae8c1457f0 legacy=0x7fae8c130010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 monclient: found mon.a 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c13a4a0 con 0x7fae8c13f650 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:26.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae93d7b640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae93d7b640 10 monclient: _renew_subs 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae93d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c143bb0 con 0x7fae8c13f650 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.219+0000 7fae93d7b640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae93d7b640 10 monclient: _renew_subs 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae93d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c13d2c0 con 0x7fae8c13f650 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae93d7b640 1 librados: init done 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae88004bf0 con 0x7fae8c13f650 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fae88004d60 con 0x7fae8c13f650 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae827fc640 10 monclient: dump: 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:26.225 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.220+0000 7fae83fff640 4 set_mon_vals no callback set 2024-03-29T11:38:26.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.221+0000 7fae93d7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:26.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.221+0000 7fae93d7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:26.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.221+0000 7fae93d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.221+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae74005080 con 0x7fae8c13f650 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.224+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fae8801ad30 con 0x7fae8c13f650 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.225+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fae88084960 con 0x7fae8c13f650 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.225+0000 7fae827fc640 10 client.4443.objecter ms_dispatch 0x7fae8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.225+0000 7fae827fc640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.225+0000 7fae827fc640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.225+0000 7fae827fc640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.226+0000 7fae91b17640 10 monclient: get_auth_request con 0x7fae7006e220 auth_method 0 2024-03-29T11:38:26.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.226+0000 7fae827fc640 10 client.4443.objecter ms_handle_connect 0x7fae7006e220 2024-03-29T11:38:26.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.234+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae88050240 con 0x7fae8c13f650 2024-03-29T11:38:26.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.234+0000 7fae827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:26.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.234+0000 7fae827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:26.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.607+0000 7fae93d7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:26.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.607+0000 7fae93d7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:26.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.607+0000 7fae93d7b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:38:26.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.607+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae74005330 con 0x7fae8c13f650 2024-03-29T11:38:26.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.609+0000 7fae827fc640 1 -- 172.21.15.71:0/1458623259 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fae88052170 con 0x7fae8c13f650 2024-03-29T11:38:26.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.609+0000 7fae827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:26.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.609+0000 7fae827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:26.610 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:26.610 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 10 librados: watch_flush enter 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 10 librados: watch_flush exit 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 10 client.4443.objecter successfully canceled tick 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 >> v1:172.21.15.71:6816/1596514235 conn(0x7fae7006e220 legacy=0x7fae700706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 10 monclient: shutdown 2024-03-29T11:38:26.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:26.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 >> v1:172.21.15.71:6789/0 conn(0x7fae8c13f650 legacy=0x7fae8c12f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 shutdown_connections 2024-03-29T11:38:26.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 >> 172.21.15.71:0/1458623259 conn(0x7fae8c066a00 msgr2=0x7fae8c136500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:26.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 shutdown_connections 2024-03-29T11:38:26.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.613+0000 7fae93d7b640 1 -- 172.21.15.71:0/1458623259 wait complete. 2024-03-29T11:38:26.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:26.614+0000 7fae93d7b640 1 librados: shutdown 2024-03-29T11:38:26.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:33.438 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.587+0000 7ffa761f4640 10 monclient: get_monmap_and_config 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.587+0000 7ffa761f4640 10 monclient: build_initial_monmap 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.587+0000 7ffa761f4640 10 monclient: monmap: 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:33.588230+0000 2024-03-29T11:38:33.588 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:33.588230+0000 2024-03-29T11:38:33.589 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:33.589 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:33.589 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:33.589 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:33.589 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:33.589 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.588+0000 7ffa761f4640 1 Processor -- start 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.588+0000 7ffa761f4640 1 -- start start 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.588+0000 7ffa761f4640 10 monclient: init 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient(hunting): picked mon.noname-b con 0x7ffa700b4100 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient(hunting): picked mon.noname-a con 0x7ffa7013ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient(hunting): picked mon.noname-c con 0x7ffa700af900 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700afd10 con 0x7ffa7013ee90 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700af6f0 con 0x7ffa700af900 2024-03-29T11:38:33.590 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700aac80 con 0x7ffa700b4100 2024-03-29T11:38:33.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:33.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa761f4640 10 monclient(hunting): authenticate will time out at 1167.743505s 2024-03-29T11:38:33.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa6ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ffa700af900 0x7ffa7014ac10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:50208/0 (socket says 172.21.15.71:50208) 2024-03-29T11:38:33.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.589+0000 7ffa6ffff640 1 -- 172.21.15.71:0/1769853799 learned_addr learned my addr 172.21.15.71:0/1769853799 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553155686 0 0) 0x7ffa700aac80 con 0x7ffa700b4100 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58003520 con 0x7ffa700b4100 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443823957 0 0) 0x7ffa700af6f0 con 0x7ffa700af900 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:33.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.590+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa700aac80 con 0x7ffa700af900 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 873627254 0 0) 0x7ffa58003520 con 0x7ffa700b4100 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 >> v1:172.21.15.81:6789/0 conn(0x7ffa700af900 legacy=0x7ffa7014ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 >> v1:172.21.15.71:6789/0 conn(0x7ffa7013ee90 legacy=0x7ffa7013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa700aaf60 con 0x7ffa700b4100 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ffa60001690 con 0x7ffa700b4100 2024-03-29T11:38:33.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: dump: 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa761f4640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa761f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:33.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa600018b0 con 0x7ffa700b4100 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 1 -- 172.21.15.71:0/1769853799 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ffa60001c40 con 0x7ffa700b4100 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa6e7fc640 10 monclient: dump: 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:33.597 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa761f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.591+0000 7ffa761f4640 4 set_mon_vals no callback set 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 10 monclient: shutdown 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 1 -- 172.21.15.71:0/1769853799 >> v1:172.21.15.71:6790/0 conn(0x7ffa700b4100 legacy=0x7ffa700adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 1 -- 172.21.15.71:0/1769853799 shutdown_connections 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 1 -- 172.21.15.71:0/1769853799 >> 172.21.15.71:0/1769853799 conn(0x7ffa700b35e0 msgr2=0x7ffa700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 1 -- 172.21.15.71:0/1769853799 shutdown_connections 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.592+0000 7ffa761f4640 1 -- 172.21.15.71:0/1769853799 wait complete. 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.593+0000 7ffa761f4640 10 monclient: build_initial_monmap 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.593+0000 7ffa761f4640 10 monclient: monmap: 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:33.598 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:33.599 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:33.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 1 librados: starting msgr at 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 1 librados: starting objecter 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 1 Processor -- start 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 1 -- start start 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 1 librados: setting wanted keys 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 1 librados: calling monclient init 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.594+0000 7ffa761f4640 10 monclient: init 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient(hunting): picked mon.noname-c con 0x7ffa700af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient(hunting): picked mon.noname-a con 0x7ffa700b4100 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7013ee90 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700721d0 con 0x7ffa700b4100 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa70139220 con 0x7ffa7013ee90 2024-03-29T11:38:33.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa701393b0 con 0x7ffa700af900 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa761f4640 10 monclient(hunting): authenticate will time out at 1167.749651s 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa6f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7ffa700af900 0x7ffa70138b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:59340/0 (socket says 172.21.15.71:59340) 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.595+0000 7ffa6f7fe640 1 -- 172.21.15.71:0/972861779 learned_addr learned my addr 172.21.15.71:0/972861779 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 client.?.objecter ms_handle_connect 0x7ffa700af900 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 client.?.objecter ms_handle_connect 0x7ffa700b4100 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000202014 0 0) 0x7ffa701393b0 con 0x7ffa700af900 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:38:33.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c004eb0 con 0x7ffa700af900 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616155604 0 0) 0x7ffa700721d0 con 0x7ffa700b4100 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa701393b0 con 0x7ffa700b4100 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3318359865 0 0) 0x7ffa5c004eb0 con 0x7ffa700af900 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa700721d0 con 0x7ffa700af900 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ffa60001c80 con 0x7ffa700af900 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.596+0000 7ffa57fff640 10 monclient(hunting): dump: 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:33.605 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2913037525 0 0) 0x7ffa701393b0 con 0x7ffa700b4100 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa5c004eb0 con 0x7ffa700b4100 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4266558674 0 0) 0x7ffa700721d0 con 0x7ffa700af900 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 >> v1:172.21.15.81:6789/0 conn(0x7ffa7013ee90 legacy=0x7ffa701e07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 >> v1:172.21.15.71:6789/0 conn(0x7ffa700b4100 legacy=0x7ffa701d4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 monclient: found mon.c 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa70139540 con 0x7ffa700af900 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 10 monclient: _renew_subs 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa70139760 con 0x7ffa700af900 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 10 monclient: _renew_subs 2024-03-29T11:38:33.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa70141ab0 con 0x7ffa700af900 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.597+0000 7ffa761f4640 1 librados: init done 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa60003a40 con 0x7ffa700af900 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ffa60003dd0 con 0x7ffa700af900 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa57fff640 10 monclient: dump: 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.608 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.598+0000 7ffa6d7fa640 4 set_mon_vals no callback set 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.599+0000 7ffa761f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.600+0000 7ffa761f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.600+0000 7ffa761f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.600+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa50005080 con 0x7ffa700af900 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.600+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7ffa60003f30 con 0x7ffa700af900 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.601+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7ffa60084be0 con 0x7ffa700af900 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.601+0000 7ffa57fff640 10 client.4493.objecter ms_dispatch 0x7ffa70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.601+0000 7ffa57fff640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:33.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.601+0000 7ffa57fff640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:33.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.601+0000 7ffa57fff640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:38:33.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.601+0000 7ffa6effd640 10 monclient: get_auth_request con 0x7ffa5c072290 auth_method 0 2024-03-29T11:38:33.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.602+0000 7ffa57fff640 10 client.4493.objecter ms_handle_connect 0x7ffa5c072290 2024-03-29T11:38:33.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.608+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa600504c0 con 0x7ffa700af900 2024-03-29T11:38:33.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.608+0000 7ffa57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:33.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.608+0000 7ffa57fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:33.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.968+0000 7ffa761f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:33.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.968+0000 7ffa761f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:33.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.968+0000 7ffa761f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:33.969 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.968+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa50005330 con 0x7ffa700af900 2024-03-29T11:38:33.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.970+0000 7ffa57fff640 1 -- 172.21.15.71:0/972861779 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7ffa600523f0 con 0x7ffa700af900 2024-03-29T11:38:33.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.970+0000 7ffa57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:33.971 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.970+0000 7ffa57fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:33.971 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:33.971 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.973+0000 7ffa761f4640 10 librados: watch_flush enter 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.973+0000 7ffa761f4640 10 librados: watch_flush exit 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.973+0000 7ffa761f4640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.973+0000 7ffa761f4640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 >> v1:172.21.15.71:6816/1596514235 conn(0x7ffa5c072290 legacy=0x7ffa5c074750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 10 monclient: shutdown 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 >> v1:172.21.15.71:6790/0 conn(0x7ffa700af900 legacy=0x7ffa70138b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.975 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 shutdown_connections 2024-03-29T11:38:33.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 >> 172.21.15.71:0/972861779 conn(0x7ffa700b35e0 msgr2=0x7ffa7013b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:33.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 shutdown_connections 2024-03-29T11:38:33.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.974+0000 7ffa761f4640 1 -- 172.21.15.71:0/972861779 wait complete. 2024-03-29T11:38:33.976 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:33.975+0000 7ffa761f4640 1 librados: shutdown 2024-03-29T11:38:33.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:40.797 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:40.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.929+0000 7febcac8d640 10 monclient: get_monmap_and_config 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.929+0000 7febcac8d640 10 monclient: build_initial_monmap 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.929+0000 7febcac8d640 10 monclient: monmap: 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:40.931119+0000 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:40.931119+0000 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:40.931 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:40.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.930+0000 7febcac8d640 1 Processor -- start 2024-03-29T11:38:40.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.930+0000 7febcac8d640 1 -- start start 2024-03-29T11:38:40.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.930+0000 7febcac8d640 10 monclient: init 2024-03-29T11:38:40.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.931+0000 7febcac8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:40.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.931+0000 7febcac8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:40.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.931+0000 7febcac8d640 10 monclient(hunting): picked mon.noname-c con 0x7febc41457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 10 monclient(hunting): picked mon.noname-a con 0x7febc413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 10 monclient(hunting): picked mon.noname-b con 0x7febc40a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40aaef0 con 0x7febc413f650 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc4145bd0 con 0x7febc41457f0 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc4145eb0 con 0x7febc40a9d70 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febcac8d640 10 monclient(hunting): authenticate will time out at 1175.087191s 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc922a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7febc40a9d70 0x7febc414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39916/0 (socket says 172.21.15.71:39916) 2024-03-29T11:38:40.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc922a640 1 -- 172.21.15.71:0/2331396273 learned_addr learned my addr 172.21.15.71:0/2331396273 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641367137 0 0) 0x7febc4145eb0 con 0x7febc40a9d70 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba4003520 con 0x7febc40a9d70 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3956275892 0 0) 0x7feba4003520 con 0x7febc40a9d70 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 >> v1:172.21.15.81:6789/0 conn(0x7febc41457f0 legacy=0x7febc413ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:38:40.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.932+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 >> v1:172.21.15.71:6789/0 conn(0x7febc413f650 legacy=0x7febc40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:40.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:40.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 1 monclient: found mon.noname-b 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc4146340 con 0x7febc40a9d70 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7febb4001670 con 0x7febc40a9d70 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: dump: 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.937 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febcac8d640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febcac8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febb40018d0 con 0x7febc40a9d70 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.933+0000 7febc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febcac8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febcac8d640 4 set_mon_vals no callback set 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febc37fe640 1 -- 172.21.15.71:0/2331396273 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7febb4001d20 con 0x7febc40a9d70 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:40.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febc37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febc37fe640 10 monclient: dump: 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febcac8d640 10 monclient: shutdown 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febcac8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.934+0000 7febcac8d640 1 -- 172.21.15.71:0/2331396273 >> v1:172.21.15.71:6790/0 conn(0x7febc40a9d70 legacy=0x7febc414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.935+0000 7febcac8d640 1 -- 172.21.15.71:0/2331396273 shutdown_connections 2024-03-29T11:38:40.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.935+0000 7febcac8d640 1 -- 172.21.15.71:0/2331396273 >> 172.21.15.71:0/2331396273 conn(0x7febc4066a00 msgr2=0x7febc4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:40.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.935+0000 7febcac8d640 1 -- 172.21.15.71:0/2331396273 shutdown_connections 2024-03-29T11:38:40.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.935+0000 7febcac8d640 1 -- 172.21.15.71:0/2331396273 wait complete. 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 10 monclient: build_initial_monmap 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 10 monclient: monmap: 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 1 librados: starting msgr at 2024-03-29T11:38:40.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 1 librados: starting objecter 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 1 Processor -- start 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 1 -- start start 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 1 librados: setting wanted keys 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 1 librados: calling monclient init 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.936+0000 7febcac8d640 10 monclient: init 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.937+0000 7febcac8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 10 monclient(hunting): picked mon.noname-c con 0x7febc40a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febc8a29640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7febc40a9d70 0x7febc4137d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:40202/0 (socket says 172.21.15.71:40202) 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 10 monclient(hunting): picked mon.noname-a con 0x7febc413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febc8a29640 1 -- 172.21.15.71:0/1530739451 learned_addr learned my addr 172.21.15.71:0/1530739451 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 10 monclient(hunting): picked mon.noname-b con 0x7febc41457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc413fda0 con 0x7febc413f650 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40ad050 con 0x7febc41457f0 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40ad330 con 0x7febc40a9d70 2024-03-29T11:38:40.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febcac8d640 10 monclient(hunting): authenticate will time out at 1175.093903s 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febc17fa640 10 client.?.objecter ms_handle_connect 0x7febc40a9d70 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.938+0000 7febc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043706428 0 0) 0x7febc40ad330 con 0x7febc40a9d70 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8004eb0 con 0x7febc40a9d70 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 client.?.objecter ms_handle_connect 0x7febc41457f0 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2059352724 0 0) 0x7feba8004eb0 con 0x7febc40a9d70 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc40ad330 con 0x7febc40a9d70 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7febbc0017d0 con 0x7febc40a9d70 2024-03-29T11:38:40.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): dump: 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606205041 0 0) 0x7febc40ad050 con 0x7febc41457f0 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:40.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8004eb0 con 0x7febc41457f0 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2727350633 0 0) 0x7febc40ad330 con 0x7febc40a9d70 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 >> v1:172.21.15.81:6789/0 conn(0x7febc41457f0 legacy=0x7febc41e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.939+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 >> v1:172.21.15.71:6789/0 conn(0x7febc413f650 legacy=0x7febc41d7370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 1 monclient: found mon.c 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40ac1c0 con 0x7febc40a9d70 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 10 monclient: _renew_subs 2024-03-29T11:38:40.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc40abae0 con 0x7febc40a9d70 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 10 monclient: _renew_subs 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc4138630 con 0x7febc40a9d70 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febcac8d640 1 librados: init done 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febbc001d60 con 0x7febc40a9d70 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7febbc004a00 con 0x7febc40a9d70 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc17fa640 10 monclient: dump: 2024-03-29T11:38:40.949 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.941+0000 7febc2ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:40.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.942+0000 7febcac8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.943+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7febbc01a9d0 con 0x7febc40a9d70 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.943+0000 7febcac8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.943+0000 7febcac8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.943+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c005080 con 0x7febc40a9d70 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7febbc0849c0 con 0x7febc40a9d70 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc17fa640 10 client.4499.objecter ms_dispatch 0x7febc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc17fa640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc17fa640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:40.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc17fa640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T11:38:40.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc3fff640 10 monclient: get_auth_request con 0x7feba806dde0 auth_method 0 2024-03-29T11:38:40.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.944+0000 7febc17fa640 10 client.4499.objecter ms_handle_connect 0x7feba806dde0 2024-03-29T11:38:40.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.950+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febbc0502a0 con 0x7febc40a9d70 2024-03-29T11:38:40.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.950+0000 7febc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:40.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:40.950+0000 7febc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:41.262 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febcac8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:41.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febcac8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:41.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febcac8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:41.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb9c005330 con 0x7febc40a9d70 2024-03-29T11:38:41.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febc17fa640 1 -- 172.21.15.71:0/1530739451 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7febbc0521d0 con 0x7febc40a9d70 2024-03-29T11:38:41.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:41.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.261+0000 7febc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:41.264 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:41.264 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:41.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 10 librados: watch_flush enter 2024-03-29T11:38:41.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 10 librados: watch_flush exit 2024-03-29T11:38:41.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T11:38:41.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 10 client.4499.objecter successfully canceled tick 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 >> v1:172.21.15.71:6816/1596514235 conn(0x7feba806dde0 legacy=0x7feba80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 10 monclient: shutdown 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.265+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 >> v1:172.21.15.71:6790/0 conn(0x7febc40a9d70 legacy=0x7febc4137d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.266+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 shutdown_connections 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.266+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 >> 172.21.15.71:0/1530739451 conn(0x7febc4066a00 msgr2=0x7febc40b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.266+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 shutdown_connections 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.266+0000 7febcac8d640 1 -- 172.21.15.71:0/1530739451 wait complete. 2024-03-29T11:38:41.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:41.266+0000 7febcac8d640 1 librados: shutdown 2024-03-29T11:38:41.281 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:48.088 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.221+0000 7f1e5e78f640 10 monclient: get_monmap_and_config 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.221+0000 7f1e5e78f640 10 monclient: build_initial_monmap 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.221+0000 7f1e5e78f640 10 monclient: monmap: 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:48.222399+0000 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:48.222399+0000 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:48.222 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:48.223 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:48.223 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:48.223 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:48.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.222+0000 7f1e5e78f640 1 Processor -- start 2024-03-29T11:38:48.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.222+0000 7f1e5e78f640 1 -- start start 2024-03-29T11:38:48.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.222+0000 7f1e5e78f640 10 monclient: init 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 10 monclient(hunting): picked mon.noname-a con 0x7f1e580ae8c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 10 monclient(hunting): picked mon.noname-b con 0x7f1e581422a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 10 monclient(hunting): picked mon.noname-c con 0x7f1e5813c600 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e580ad5d0 con 0x7f1e580ae8c0 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.223+0000 7f1e5e78f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e580acca0 con 0x7f1e5813c600 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e5cd2c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1e5813c600 0x7f1e5814a640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:34180/0 (socket says 172.21.15.71:34180) 2024-03-29T11:38:48.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e5cd2c640 1 -- 172.21.15.71:0/2794032125 learned_addr learned my addr 172.21.15.71:0/2794032125 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2794032125 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e580ae1c0 con 0x7f1e581422a0 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e5e78f640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e5e78f640 10 monclient(hunting): authenticate will time out at 1182.378192s 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049322131 0 0) 0x7f1e580acca0 con 0x7f1e5813c600 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e40005730 con 0x7f1e5813c600 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474091106 0 0) 0x7f1e580ae1c0 con 0x7f1e581422a0 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e580acca0 con 0x7f1e581422a0 2024-03-29T11:38:48.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2279148587 0 0) 0x7f1e40005730 con 0x7f1e5813c600 2024-03-29T11:38:48.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 >> v1:172.21.15.71:6790/0 conn(0x7f1e581422a0 legacy=0x7f1e5813bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.224+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 >> v1:172.21.15.71:6789/0 conn(0x7f1e580ae8c0 legacy=0x7f1e580adab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:48.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:48.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 1 monclient: found mon.noname-c 2024-03-29T11:38:48.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:48.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e580ae350 con 0x7f1e5813c600 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1e50001a10 con 0x7f1e5813c600 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: dump: 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:48.229 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e5e78f640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e5e78f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e50001b70 con 0x7f1e5813c600 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 1 -- 172.21.15.71:0/2794032125 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1e50001f00 con 0x7f1e5813c600 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e56ffd640 10 monclient: dump: 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:48.230 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e5e78f640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:48.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.227+0000 7f1e5e78f640 4 set_mon_vals no callback set 2024-03-29T11:38:48.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 10 monclient: shutdown 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2794032125 >> v1:172.21.15.81:6789/0 conn(0x7f1e5813c600 legacy=0x7f1e5814a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2794032125 shutdown_connections 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2794032125 >> 172.21.15.71:0/2794032125 conn(0x7f1e580b3c50 msgr2=0x7f1e580b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2794032125 shutdown_connections 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.228+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2794032125 wait complete. 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.229+0000 7f1e5e78f640 10 monclient: build_initial_monmap 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.229+0000 7f1e5e78f640 10 monclient: monmap: 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:48.233 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.230+0000 7f1e5e78f640 1 librados: starting msgr at 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.230+0000 7f1e5e78f640 1 librados: starting objecter 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.230+0000 7f1e5e78f640 1 Processor -- start 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.231+0000 7f1e5e78f640 1 -- start start 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.231+0000 7f1e5e78f640 1 librados: setting wanted keys 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.231+0000 7f1e5e78f640 1 librados: calling monclient init 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.231+0000 7f1e5e78f640 10 monclient: init 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient(hunting): picked mon.noname-b con 0x7f1e580ae8c0 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:48.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient(hunting): picked mon.noname-a con 0x7f1e5813c600 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:48.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient(hunting): picked mon.noname-c con 0x7f1e581422a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:48.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5813cbb0 con 0x7f1e5813c600 2024-03-29T11:38:48.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e5813d040 con 0x7f1e580ae8c0 2024-03-29T11:38:48.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e581338e0 con 0x7f1e581422a0 2024-03-29T11:38:48.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:48.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5e78f640 10 monclient(hunting): authenticate will time out at 1182.386439s 2024-03-29T11:38:48.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5cd2c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f1e581422a0 0x7f1e581331d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50924/0 (socket says 172.21.15.71:50924) 2024-03-29T11:38:48.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.232+0000 7f1e5cd2c640 1 -- 172.21.15.71:0/2468382775 learned_addr learned my addr 172.21.15.71:0/2468382775 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 10 client.?.objecter ms_handle_connect 0x7f1e581422a0 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130567460 0 0) 0x7f1e581338e0 con 0x7f1e581422a0 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e44004eb0 con 0x7f1e581422a0 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209414078 0 0) 0x7f1e44004eb0 con 0x7f1e581422a0 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e581338e0 con 0x7f1e581422a0 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.233+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1e50001e80 con 0x7f1e581422a0 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:48.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1375224186 0 0) 0x7f1e581338e0 con 0x7f1e581422a0 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 >> v1:172.21.15.81:6789/0 conn(0x7f1e580ae8c0 legacy=0x7f1e5813d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.234+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 >> v1:172.21.15.71:6789/0 conn(0x7f1e5813c600 legacy=0x7f1e58132ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 1 monclient: found mon.c 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e58072bf0 con 0x7f1e581422a0 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e5e78f640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e5e78f640 10 monclient: _renew_subs 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e5e78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.235+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e58072150 con 0x7f1e581422a0 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.236+0000 7f1e5e78f640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:48.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.236+0000 7f1e5e78f640 10 monclient: _renew_subs 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.236+0000 7f1e5e78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.236+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e580b38d0 con 0x7f1e581422a0 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.236+0000 7f1e5e78f640 1 librados: init done 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.237+0000 7f1e5e78f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.237+0000 7f1e5e78f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.237+0000 7f1e5e78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.237+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e580724a0 con 0x7f1e581422a0 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.237+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e50004e10 con 0x7f1e581422a0 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.237+0000 7f1e54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.238+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f1e50005220 con 0x7f1e581422a0 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.238+0000 7f1e54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.238+0000 7f1e54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.238+0000 7f1e54ff9640 10 monclient: dump: 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:48.243 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.238+0000 7f1e567fc640 4 set_mon_vals no callback set 2024-03-29T11:38:48.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.239+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f1e50005440 con 0x7f1e581422a0 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f1e50085030 con 0x7f1e581422a0 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e57fff640 10 monclient: get_auth_request con 0x7f1e4406dd00 auth_method 0 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e54ff9640 10 client.4505.objecter ms_dispatch 0x7f1e58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e54ff9640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e54ff9640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e54ff9640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:38:48.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.240+0000 7f1e54ff9640 10 client.4505.objecter ms_handle_connect 0x7f1e4406dd00 2024-03-29T11:38:48.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.246+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e50050910 con 0x7f1e581422a0 2024-03-29T11:38:48.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.246+0000 7f1e54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:48.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.246+0000 7f1e54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:48.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.547+0000 7f1e5e78f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:48.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.547+0000 7f1e5e78f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:48.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.547+0000 7f1e5e78f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:38:48.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.547+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e5813cd60 con 0x7f1e581422a0 2024-03-29T11:38:48.548 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.548+0000 7f1e54ff9640 1 -- 172.21.15.71:0/2468382775 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f1e5813cd60 con 0x7f1e581422a0 2024-03-29T11:38:48.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.548+0000 7f1e54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:48.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.548+0000 7f1e54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:48.549 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:48.549 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.550+0000 7f1e5e78f640 10 librados: watch_flush enter 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.550+0000 7f1e5e78f640 10 librados: watch_flush exit 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.550+0000 7f1e5e78f640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.550+0000 7f1e5e78f640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 >> v1:172.21.15.71:6816/1596514235 conn(0x7f1e4406dd00 legacy=0x7f1e440701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 10 monclient: shutdown 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 >> v1:172.21.15.71:6790/0 conn(0x7f1e581422a0 legacy=0x7f1e581331d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 shutdown_connections 2024-03-29T11:38:48.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 >> 172.21.15.71:0/2468382775 conn(0x7f1e580b3c50 msgr2=0x7f1e58139280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:48.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 shutdown_connections 2024-03-29T11:38:48.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 -- 172.21.15.71:0/2468382775 wait complete. 2024-03-29T11:38:48.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:48.551+0000 7f1e5e78f640 1 librados: shutdown 2024-03-29T11:38:48.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:55.372 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.521+0000 7f6639296640 10 monclient: get_monmap_and_config 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.521+0000 7f6639296640 10 monclient: build_initial_monmap 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.521+0000 7f6639296640 10 monclient: monmap: 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:38:55.522317+0000 2024-03-29T11:38:55.522 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:38:55.522317+0000 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.522+0000 7f6639296640 1 Processor -- start 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.522+0000 7f6639296640 1 -- start start 2024-03-29T11:38:55.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.522+0000 7f6639296640 10 monclient: init 2024-03-29T11:38:55.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:55.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:55.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient(hunting): picked mon.noname-c con 0x7f66340b1f60 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient(hunting): picked mon.noname-b con 0x7f6634071eb0 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient(hunting): picked mon.noname-a con 0x7f6634143d10 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663406ef80 con 0x7f6634143d10 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663406f260 con 0x7f66340b1f60 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66340b40c0 con 0x7f6634071eb0 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.523+0000 7f6639296640 10 monclient(hunting): authenticate will time out at 1189.677237s 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.525+0000 7f66327fc640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f6634071eb0 0x7f6634146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:38172/0 (socket says 172.21.15.71:38172) 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.525+0000 7f66327fc640 1 -- 172.21.15.71:0/4071323514 learned_addr learned my addr 172.21.15.71:0/4071323514 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852666580 0 0) 0x7f663406ef80 con 0x7f6634143d10 2024-03-29T11:38:55.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f661c003520 con 0x7f6634143d10 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344339420 0 0) 0x7f663406f260 con 0x7f66340b1f60 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663406ef80 con 0x7f66340b1f60 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828651672 0 0) 0x7f66340b40c0 con 0x7f6634071eb0 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.526+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663406f260 con 0x7f6634071eb0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2009472633 0 0) 0x7f663406f260 con 0x7f6634071eb0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 >> v1:172.21.15.81:6789/0 conn(0x7f66340b1f60 legacy=0x7f6634143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 >> v1:172.21.15.71:6789/0 conn(0x7f6634143d10 legacy=0x7f6634146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 1 monclient: found mon.noname-b 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634155d70 con 0x7f6634071eb0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f662c001930 con 0x7f6634071eb0 2024-03-29T11:38:55.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6639296640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6631ffb640 10 monclient: dump: 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:55.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.527+0000 7f6639296640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.528+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f662c001a90 con 0x7f6634071eb0 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.528+0000 7f6631ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.528+0000 7f6639296640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.528+0000 7f6639296640 4 set_mon_vals no callback set 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6631ffb640 1 -- 172.21.15.71:0/4071323514 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f662c001ee0 con 0x7f6634071eb0 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6631ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6631ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6631ffb640 10 monclient: dump: 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:55.536 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6639296640 10 monclient: shutdown 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6639296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6639296640 1 -- 172.21.15.71:0/4071323514 >> v1:172.21.15.71:6790/0 conn(0x7f6634071eb0 legacy=0x7f6634146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6639296640 1 -- 172.21.15.71:0/4071323514 shutdown_connections 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.529+0000 7f6639296640 1 -- 172.21.15.71:0/4071323514 >> 172.21.15.71:0/4071323514 conn(0x7f6634066a00 msgr2=0x7f6634066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.530+0000 7f6639296640 1 -- 172.21.15.71:0/4071323514 shutdown_connections 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.530+0000 7f6639296640 1 -- 172.21.15.71:0/4071323514 wait complete. 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.531+0000 7f6639296640 10 monclient: build_initial_monmap 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.531+0000 7f6639296640 10 monclient: monmap: 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:38:55.537 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:55.538 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:55.538 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:38:55.538 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:38:55.538 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:38:55.538 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.531+0000 7f6639296640 1 librados: starting msgr at 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.531+0000 7f6639296640 1 librados: starting objecter 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.531+0000 7f6639296640 1 Processor -- start 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 1 -- start start 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 1 librados: setting wanted keys 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 1 librados: calling monclient init 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient: init 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:55.542 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient(hunting): picked mon.noname-a con 0x7f66340b1f60 addr v1:172.21.15.71:6789/0 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient(hunting): picked mon.noname-c con 0x7f6634143d10 addr v1:172.21.15.71:6790/0 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient(hunting): picked mon.noname-b con 0x7f6634134730 addr v1:172.21.15.81:6789/0 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66341425f0 con 0x7f66340b1f60 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66341428d0 con 0x7f6634134730 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6634141830 con 0x7f6634143d10 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.532+0000 7f6639296640 10 monclient(hunting): authenticate will time out at 1189.686858s 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.535+0000 7f66337fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f6634134730 0x7f66341e7100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:45407/0 (socket says 172.21.15.71:45407) 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f66337fe640 1 -- 172.21.15.71:0/732507467 learned_addr learned my addr 172.21.15.71:0/732507467 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:38:55.543 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 client.?.objecter ms_handle_connect 0x7f6634134730 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 client.?.objecter ms_handle_connect 0x7f6634143d10 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946110564 0 0) 0x7f66341428d0 con 0x7f6634134730 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620004eb0 con 0x7f6634134730 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892128167 0 0) 0x7f6634141830 con 0x7f6634143d10 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.536+0000 7f661b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66341428d0 con 0x7f6634143d10 2024-03-29T11:38:55.544 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3506717456 0 0) 0x7f6620004eb0 con 0x7f6634134730 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6634141830 con 0x7f6634134730 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f66280018b0 con 0x7f6634134730 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:55.545 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.537+0000 7f661b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:55.546 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.538+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 458587408 0 0) 0x7f6634141830 con 0x7f6634134730 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.538+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 >> v1:172.21.15.71:6790/0 conn(0x7f6634143d10 legacy=0x7f6634070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.538+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 >> v1:172.21.15.71:6789/0 conn(0x7f66340b1f60 legacy=0x7f6634141120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 1 monclient: found mon.b 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634133af0 con 0x7f6634134730 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:55.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f661b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 10 monclient: _renew_subs 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6634142cf0 con 0x7f6634134730 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 10 monclient: _renew_subs 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6634136c20 con 0x7f6634134730 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.539+0000 7f6639296640 1 librados: init done 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.540+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66280035f0 con 0x7f6634134730 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.540+0000 7f661b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.540+0000 7f66317fa640 4 set_mon_vals no callback set 2024-03-29T11:38:55.550 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.540+0000 7f6639296640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6628001a10 con 0x7f6634134730 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f6639296640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f6639296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6614005080 con 0x7f6634134730 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f661b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f661b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.541+0000 7f661b7fe640 10 monclient: dump: 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:38:55.551 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.542+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f6628003ac0 con 0x7f6634134730 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f6628084950 con 0x7f6634134730 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f661b7fe640 10 client.4477.objecter ms_dispatch 0x7f6634000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f661b7fe640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f661b7fe640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f661b7fe640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f6632ffd640 10 monclient: get_auth_request con 0x7f662006dd00 auth_method 0 2024-03-29T11:38:55.552 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.543+0000 7f661b7fe640 10 client.4477.objecter ms_handle_connect 0x7f662006dd00 2024-03-29T11:38:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.550+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6628050230 con 0x7f6634134730 2024-03-29T11:38:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.550+0000 7f661b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:55.553 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.550+0000 7f661b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:55.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.865+0000 7f6639296640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:55.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.865+0000 7f6639296640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:55.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.865+0000 7f6639296640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:38:55.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.865+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6614005330 con 0x7f6634134730 2024-03-29T11:38:55.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.866+0000 7f661b7fe640 1 -- 172.21.15.71:0/732507467 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f6628052160 con 0x7f6634134730 2024-03-29T11:38:55.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.866+0000 7f661b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:55.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.866+0000 7f661b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:55.869 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:38:55.869 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.869+0000 7f6639296640 10 librados: watch_flush enter 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 10 librados: watch_flush exit 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 >> v1:172.21.15.71:6816/1596514235 conn(0x7f662006dd00 legacy=0x7f66200701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 10 monclient: shutdown 2024-03-29T11:38:55.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:55.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 >> v1:172.21.15.81:6789/0 conn(0x7f6634134730 legacy=0x7f66341e7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 shutdown_connections 2024-03-29T11:38:55.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 >> 172.21.15.71:0/732507467 conn(0x7f6634066a00 msgr2=0x7f6634146af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:55.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 shutdown_connections 2024-03-29T11:38:55.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 -- 172.21.15.71:0/732507467 wait complete. 2024-03-29T11:38:55.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:38:55.870+0000 7f6639296640 1 librados: shutdown 2024-03-29T11:38:55.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:02.693 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:02.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.840+0000 7f577e50b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:02.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.840+0000 7f577e50b640 10 monclient: build_initial_monmap 2024-03-29T11:39:02.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.840+0000 7f577e50b640 10 monclient: monmap: 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:02.841425+0000 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:02.841425+0000 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:02.842 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.842+0000 7f577e50b640 1 Processor -- start 2024-03-29T11:39:02.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.842+0000 7f577e50b640 1 -- start start 2024-03-29T11:39:02.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.842+0000 7f577e50b640 10 monclient: init 2024-03-29T11:39:02.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:02.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:02.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient(hunting): picked mon.noname-c con 0x7f5770006890 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient(hunting): picked mon.noname-b con 0x7f5770006c70 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient(hunting): picked mon.noname-a con 0x7f577002b4c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577002c830 con 0x7f577002b4c0 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577002afa0 con 0x7f5770006890 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577002aba0 con 0x7f5770006c70 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:02.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.843+0000 7f577e50b640 10 monclient(hunting): authenticate will time out at 1196.997748s 2024-03-29T11:39:02.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.846+0000 7f577cd08640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5770006c70 0x7f57700c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53010/0 (socket says 172.21.15.71:53010) 2024-03-29T11:39:02.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.846+0000 7f577cd08640 1 -- 172.21.15.71:0/585440558 learned_addr learned my addr 172.21.15.71:0/585440558 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.847+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051633267 0 0) 0x7f577002c830 con 0x7f577002b4c0 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.847+0000 7f576ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.847+0000 7f576ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.847+0000 7f576ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.847+0000 7f576ffff640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.847+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5764003520 con 0x7f577002b4c0 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4134406264 0 0) 0x7f5764003520 con 0x7f577002b4c0 2024-03-29T11:39:02.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 >> v1:172.21.15.71:6790/0 conn(0x7f5770006c70 legacy=0x7f57700c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 >> v1:172.21.15.81:6789/0 conn(0x7f5770006890 legacy=0x7f57700c0a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 monclient: found mon.noname-a 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577002ad30 con 0x7f577002b4c0 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: _finish_auth 0 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f577804f730 con 0x7f577002b4c0 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: dump: 2024-03-29T11:39:02.850 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f577e50b640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f577e50b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57780008d0 con 0x7f577002b4c0 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f577e50b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:02.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f577e50b640 4 set_mon_vals no callback set 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 1 -- 172.21.15.71:0/585440558 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f57780758b0 con 0x7f577002b4c0 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.848+0000 7f576ffff640 10 monclient: dump: 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:02.852 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 10 monclient: shutdown 2024-03-29T11:39:02.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:02.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 1 -- 172.21.15.71:0/585440558 >> v1:172.21.15.71:6789/0 conn(0x7f577002b4c0 legacy=0x7f57700cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:02.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 1 -- 172.21.15.71:0/585440558 shutdown_connections 2024-03-29T11:39:02.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 1 -- 172.21.15.71:0/585440558 >> 172.21.15.71:0/585440558 conn(0x7f57700058f0 msgr2=0x7f5770005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:02.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 1 -- 172.21.15.71:0/585440558 shutdown_connections 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.849+0000 7f577e50b640 1 -- 172.21.15.71:0/585440558 wait complete. 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.850+0000 7f577e50b640 10 monclient: build_initial_monmap 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.850+0000 7f577e50b640 10 monclient: monmap: 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 1 librados: starting msgr at 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 1 librados: starting objecter 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 1 Processor -- start 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 1 -- start start 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 1 librados: setting wanted keys 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 1 librados: calling monclient init 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 10 monclient: init 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.851+0000 7f577e50b640 10 monclient(hunting): picked mon.noname-c con 0x7f5770006890 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.852+0000 7f577d509640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5770006890 0x7f577014d850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53552/0 (socket says 172.21.15.71:53552) 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 10 monclient(hunting): picked mon.noname-b con 0x7f577002b4c0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577d509640 1 -- 172.21.15.71:0/1354548611 learned_addr learned my addr 172.21.15.71:0/1354548611 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 10 monclient(hunting): picked mon.noname-a con 0x7f57700c6e30 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577014d3c0 con 0x7f57700c6e30 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700bd6a0 con 0x7f577002b4c0 2024-03-29T11:39:02.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57700bd910 con 0x7f5770006890 2024-03-29T11:39:02.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:02.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f577e50b640 10 monclient(hunting): authenticate will time out at 1197.007774s 2024-03-29T11:39:02.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f576dffb640 10 client.?.objecter ms_handle_connect 0x7f5770006890 2024-03-29T11:39:02.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.853+0000 7f576dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:02.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392822080 0 0) 0x7f57700bd910 con 0x7f5770006890 2024-03-29T11:39:02.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:02.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:02.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f575c004eb0 con 0x7f5770006890 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 client.?.objecter ms_handle_connect 0x7f577002b4c0 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746826950 0 0) 0x7f57700bd6a0 con 0x7f577002b4c0 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.856+0000 7f576dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.857+0000 7f576dffb640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:39:02.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.857+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57700bd910 con 0x7f577002b4c0 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321485560 0 0) 0x7f57700bd910 con 0x7f577002b4c0 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57700bd6a0 con 0x7f577002b4c0 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4087258438 0 0) 0x7f575c004eb0 con 0x7f5770006890 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57700bd910 con 0x7f5770006890 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f57680018a0 con 0x7f577002b4c0 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): dump: 2024-03-29T11:39:02.865 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5774001750 con 0x7f5770006890 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.860+0000 7f576dffb640 10 monclient(hunting): dump: 2024-03-29T11:39:02.866 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3454257171 0 0) 0x7f57700bd6a0 con 0x7f577002b4c0 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 >> v1:172.21.15.71:6790/0 conn(0x7f5770006890 legacy=0x7f577014d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 >> v1:172.21.15.71:6789/0 conn(0x7f57700c6e30 legacy=0x7f577016bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 monclient: found mon.b 2024-03-29T11:39:02.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57700bdbf0 con 0x7f577002b4c0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 10 monclient: _renew_subs 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57700bded0 con 0x7f577002b4c0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 10 monclient: _renew_subs 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f577016d260 con 0x7f577002b4c0 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f577e50b640 1 librados: init done 2024-03-29T11:39:02.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5768001a00 con 0x7f577002b4c0 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f57680049c0 con 0x7f577002b4c0 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.861+0000 7f576dffb640 10 monclient: dump: 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:02.869 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:02.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.862+0000 7f576effd640 4 set_mon_vals no callback set 2024-03-29T11:39:02.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.863+0000 7f577e50b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:02.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.863+0000 7f577e50b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:02.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.864+0000 7f577e50b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:02.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.864+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5758005080 con 0x7f577002b4c0 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.865+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f5768005c10 con 0x7f577002b4c0 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.866+0000 7f577d509640 10 monclient: get_auth_request con 0x7f575c06ded0 auth_method 0 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.867+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f57680848f0 con 0x7f577002b4c0 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.867+0000 7f576dffb640 10 client.4480.objecter ms_dispatch 0x7f5778000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.867+0000 7f576dffb640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.867+0000 7f576dffb640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.867+0000 7f576dffb640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T11:39:02.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.867+0000 7f576dffb640 10 client.4480.objecter ms_handle_connect 0x7f575c06ded0 2024-03-29T11:39:02.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.876+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5768050000 con 0x7f577002b4c0 2024-03-29T11:39:02.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.876+0000 7f576dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:02.877 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:02.876+0000 7f576dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:03.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.190+0000 7f577e50b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:03.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.190+0000 7f577e50b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:03.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.190+0000 7f577e50b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:03.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.190+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5758005330 con 0x7f577002b4c0 2024-03-29T11:39:03.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.192+0000 7f576dffb640 1 -- 172.21.15.71:0/1354548611 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f5768052100 con 0x7f577002b4c0 2024-03-29T11:39:03.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.192+0000 7f576dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:03.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.192+0000 7f576dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:03.195 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:03.195 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:03.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.196+0000 7f577e50b640 10 librados: watch_flush enter 2024-03-29T11:39:03.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.196+0000 7f577e50b640 10 librados: watch_flush exit 2024-03-29T11:39:03.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.196+0000 7f577e50b640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T11:39:03.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.196+0000 7f577e50b640 10 client.4480.objecter successfully canceled tick 2024-03-29T11:39:03.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 >> v1:172.21.15.71:6816/1596514235 conn(0x7f575c06ded0 legacy=0x7f575c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 10 monclient: shutdown 2024-03-29T11:39:03.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:03.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 >> v1:172.21.15.81:6789/0 conn(0x7f577002b4c0 legacy=0x7f577014df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 shutdown_connections 2024-03-29T11:39:03.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 >> 172.21.15.71:0/1354548611 conn(0x7f57700058f0 msgr2=0x7f57700c0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:03.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 shutdown_connections 2024-03-29T11:39:03.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 -- 172.21.15.71:0/1354548611 wait complete. 2024-03-29T11:39:03.199 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:03.197+0000 7f577e50b640 1 librados: shutdown 2024-03-29T11:39:03.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:10.021 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:10.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.170+0000 7f34b589e640 10 monclient: get_monmap_and_config 2024-03-29T11:39:10.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.170+0000 7f34b589e640 10 monclient: build_initial_monmap 2024-03-29T11:39:10.171 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.170+0000 7f34b589e640 10 monclient: monmap: 2024-03-29T11:39:10.171 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:10.171 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:10.171 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:10.171395+0000 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:10.171395+0000 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:10.172 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:10.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.172+0000 7f34b589e640 1 Processor -- start 2024-03-29T11:39:10.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.172+0000 7f34b589e640 1 -- start start 2024-03-29T11:39:10.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.172+0000 7f34b589e640 10 monclient: init 2024-03-29T11:39:10.174 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient(hunting): picked mon.noname-a con 0x7f34b00b04e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient(hunting): picked mon.noname-c con 0x7f34b006f0b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient(hunting): picked mon.noname-b con 0x7f34b00728b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b006fcf0 con 0x7f34b00b04e0 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b013b740 con 0x7f34b006f0b0 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b013b8d0 con 0x7f34b00728b0 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:10.175 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.173+0000 7f34b589e640 10 monclient(hunting): authenticate will time out at 1204.327701s 2024-03-29T11:39:10.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.175+0000 7f34b509d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f34b00728b0 0x7f34b00743e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:45456/0 (socket says 172.21.15.71:45456) 2024-03-29T11:39:10.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.175+0000 7f34b509d640 1 -- 172.21.15.71:0/3833151122 learned_addr learned my addr 172.21.15.71:0/3833151122 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:10.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262731450 0 0) 0x7f34b013b8d0 con 0x7f34b00728b0 2024-03-29T11:39:10.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:39:10.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3498003520 con 0x7f34b00728b0 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133083652 0 0) 0x7f34b006fcf0 con 0x7f34b00b04e0 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3498003840 con 0x7f34b00b04e0 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033061786 0 0) 0x7f34b013b740 con 0x7f34b006f0b0 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3498003b80 con 0x7f34b006f0b0 2024-03-29T11:39:10.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181278113 0 0) 0x7f3498003520 con 0x7f34b00728b0 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 >> v1:172.21.15.81:6789/0 conn(0x7f34b006f0b0 legacy=0x7f34b00af950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 >> v1:172.21.15.71:6789/0 conn(0x7f34b00b04e0 legacy=0x7f34b0071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.177+0000 7f34af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b013ba60 con 0x7f34b00728b0 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f34a0001670 con 0x7f34b00728b0 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:10.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: dump: 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34a0001850 con 0x7f34b00728b0 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 1 -- 172.21.15.71:0/3833151122 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f34a0001c60 con 0x7f34b00728b0 2024-03-29T11:39:10.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.178+0000 7f34af7fe640 10 monclient: dump: 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:10.184 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.180+0000 7f34b589e640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.180+0000 7f34b589e640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.180+0000 7f34b589e640 4 set_mon_vals no callback set 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 10 monclient: shutdown 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 1 -- 172.21.15.71:0/3833151122 >> v1:172.21.15.71:6790/0 conn(0x7f34b00728b0 legacy=0x7f34b00743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 1 -- 172.21.15.71:0/3833151122 shutdown_connections 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 1 -- 172.21.15.71:0/3833151122 >> 172.21.15.71:0/3833151122 conn(0x7f34b00b09d0 msgr2=0x7f34b00b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 1 -- 172.21.15.71:0/3833151122 shutdown_connections 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.181+0000 7f34b589e640 1 -- 172.21.15.71:0/3833151122 wait complete. 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.182+0000 7f34b589e640 10 monclient: build_initial_monmap 2024-03-29T11:39:10.186 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.182+0000 7f34b589e640 10 monclient: monmap: 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.182+0000 7f34b589e640 1 librados: starting msgr at 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.182+0000 7f34b589e640 1 librados: starting objecter 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.182+0000 7f34b589e640 1 Processor -- start 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.184+0000 7f34b589e640 1 -- start start 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.184+0000 7f34b589e640 1 librados: setting wanted keys 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.184+0000 7f34b589e640 1 librados: calling monclient init 2024-03-29T11:39:10.187 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.184+0000 7f34b589e640 10 monclient: init 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.184+0000 7f34b589e640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.184+0000 7f34b589e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 10 monclient(hunting): picked mon.noname-b con 0x7f34b006f0b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 10 monclient(hunting): picked mon.noname-c con 0x7f34b00728b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 10 monclient(hunting): picked mon.noname-a con 0x7f34b00b04e0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b013e4b0 con 0x7f34b00b04e0 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b013e720 con 0x7f34b006f0b0 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b013f250 con 0x7f34b00728b0 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:10.188 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34b589e640 10 monclient(hunting): authenticate will time out at 1204.339406s 2024-03-29T11:39:10.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34affff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f34b00728b0 0x7f34b014e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:52104/0 (socket says 172.21.15.71:52104) 2024-03-29T11:39:10.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.185+0000 7f34affff640 1 -- 172.21.15.71:0/2206391677 learned_addr learned my addr 172.21.15.71:0/2206391677 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:10.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 client.?.objecter ms_handle_connect 0x7f34b00728b0 2024-03-29T11:39:10.189 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:10.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261737125 0 0) 0x7f34b013f250 con 0x7f34b00728b0 2024-03-29T11:39:10.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:39:10.190 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f349c004eb0 con 0x7f34b00728b0 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 client.?.objecter ms_handle_connect 0x7f34b006f0b0 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.188+0000 7f34ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848065257 0 0) 0x7f349c004eb0 con 0x7f34b00728b0 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b013f250 con 0x7f34b00728b0 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f34a4001a30 con 0x7f34b00728b0 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:10.191 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 client.?.objecter ms_handle_connect 0x7f34b00b04e0 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243602206 0 0) 0x7f34b013e720 con 0x7f34b006f0b0 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.192 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f349c004eb0 con 0x7f34b006f0b0 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019714465 0 0) 0x7f34b013e4b0 con 0x7f34b00b04e0 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b013e720 con 0x7f34b00b04e0 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1637516816 0 0) 0x7f34b013f250 con 0x7f34b00728b0 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 >> v1:172.21.15.81:6789/0 conn(0x7f34b006f0b0 legacy=0x7f34b0141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 >> v1:172.21.15.71:6789/0 conn(0x7f34b00b04e0 legacy=0x7f34b014eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 monclient: found mon.c 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:10.193 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b013ee20 con 0x7f34b00728b0 2024-03-29T11:39:10.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:10.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:10.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:10.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34a4001b90 con 0x7f34b00728b0 2024-03-29T11:39:10.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.189+0000 7f34ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:10.194 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f34a4003a60 con 0x7f34b00728b0 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34ad7fa640 10 monclient: dump: 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:10.195 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34b589e640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34b589e640 10 monclient: _renew_subs 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34b589e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.190+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b0142440 con 0x7f34b00728b0 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.191+0000 7f34b589e640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.191+0000 7f34b589e640 10 monclient: _renew_subs 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.191+0000 7f34b589e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.191+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b01427e0 con 0x7f34b00728b0 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.191+0000 7f34b589e640 1 librados: init done 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.193+0000 7f34b589e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.193+0000 7f34b589e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.193+0000 7f34b589e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:10.196 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.193+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b013f3e0 con 0x7f34b00728b0 2024-03-29T11:39:10.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.194+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f34a4003bc0 con 0x7f34b00728b0 2024-03-29T11:39:10.197 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.195+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f34a4084be0 con 0x7f34b00728b0 2024-03-29T11:39:10.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.195+0000 7f34ad7fa640 10 client.4523.objecter ms_dispatch 0x7f34b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:10.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.195+0000 7f34ad7fa640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:10.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.195+0000 7f34ad7fa640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:10.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.195+0000 7f34ad7fa640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-29T11:39:10.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.195+0000 7f34b489c640 10 monclient: get_auth_request con 0x7f349c06de60 auth_method 0 2024-03-29T11:39:10.198 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.196+0000 7f34ad7fa640 10 client.4523.objecter ms_handle_connect 0x7f349c06de60 2024-03-29T11:39:10.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.204+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34a40504c0 con 0x7f34b00728b0 2024-03-29T11:39:10.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.204+0000 7f34ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:10.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.204+0000 7f34ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:10.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.531+0000 7f34b589e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:10.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.531+0000 7f34b589e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:10.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.531+0000 7f34b589e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:10.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.531+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34b00b4de0 con 0x7f34b00728b0 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.534+0000 7f34ad7fa640 1 -- 172.21.15.71:0/2206391677 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f34a40523f0 con 0x7f34b00728b0 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.534+0000 7f34ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:10.536 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.534+0000 7f34ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.537+0000 7f34b589e640 10 librados: watch_flush enter 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.537+0000 7f34b589e640 10 librados: watch_flush exit 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.537+0000 7f34b589e640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.537+0000 7f34b589e640 10 client.4523.objecter successfully canceled tick 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.538+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 >> v1:172.21.15.71:6816/1596514235 conn(0x7f349c06de60 legacy=0x7f349c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.538+0000 7f34b589e640 10 monclient: shutdown 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.538+0000 7f34b589e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.538+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 >> v1:172.21.15.71:6790/0 conn(0x7f34b00728b0 legacy=0x7f34b014e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.538+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 shutdown_connections 2024-03-29T11:39:10.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.538+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 >> 172.21.15.71:0/2206391677 conn(0x7f34b00b09d0 msgr2=0x7f34b00af160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:10.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.539+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 shutdown_connections 2024-03-29T11:39:10.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.539+0000 7f34b589e640 1 -- 172.21.15.71:0/2206391677 wait complete. 2024-03-29T11:39:10.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:10.540+0000 7f34b589e640 1 librados: shutdown 2024-03-29T11:39:10.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:17.367 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.514+0000 7fec7d702640 10 monclient: get_monmap_and_config 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.514+0000 7fec7d702640 10 monclient: build_initial_monmap 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.514+0000 7fec7d702640 10 monclient: monmap: 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:17.515821+0000 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:17.515821+0000 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:17.516 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:17.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:17.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.518+0000 7fec7d702640 1 Processor -- start 2024-03-29T11:39:17.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.518+0000 7fec7d702640 1 -- start start 2024-03-29T11:39:17.519 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.518+0000 7fec7d702640 10 monclient: init 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient(hunting): picked mon.noname-b con 0x7fec781457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient(hunting): picked mon.noname-c con 0x7fec7813f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient(hunting): picked mon.noname-a con 0x7fec780a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780aaef0 con 0x7fec780a9d70 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec78145bd0 con 0x7fec7813f650 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec78145eb0 con 0x7fec781457f0 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:17.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec7d702640 10 monclient(hunting): authenticate will time out at 1211.673222s 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec76ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fec781457f0 0x7fec7813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48638/0 (socket says 172.21.15.71:48638) 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.519+0000 7fec76ffd640 1 -- 172.21.15.71:0/3399286980 learned_addr learned my addr 172.21.15.71:0/3399286980 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799128345 0 0) 0x7fec78145eb0 con 0x7fec781457f0 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec60003520 con 0x7fec781457f0 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3868955289 0 0) 0x7fec60003520 con 0x7fec781457f0 2024-03-29T11:39:17.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.520+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 >> v1:172.21.15.81:6789/0 conn(0x7fec7813f650 legacy=0x7fec780b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:17.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.522+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 >> v1:172.21.15.71:6789/0 conn(0x7fec780a9d70 legacy=0x7fec7814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:17.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.522+0000 7fec75ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:17.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 1 monclient: found mon.noname-b 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec78146340 con 0x7fec781457f0 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fec70001a10 con 0x7fec781457f0 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: dump: 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.525 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec70001ca0 con 0x7fec781457f0 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 1 -- 172.21.15.71:0/3399286980 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fec70002030 con 0x7fec781457f0 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec75ffb640 10 monclient: dump: 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.526 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.527 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:17.527 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:17.527 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:17.527 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:17.527 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:17.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec7d702640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:39:17.528 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec7d702640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.523+0000 7fec7d702640 4 set_mon_vals no callback set 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.524+0000 7fec7d702640 10 monclient: shutdown 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.524+0000 7fec7d702640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.524+0000 7fec7d702640 1 -- 172.21.15.71:0/3399286980 >> v1:172.21.15.71:6790/0 conn(0x7fec781457f0 legacy=0x7fec7813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.525+0000 7fec7d702640 1 -- 172.21.15.71:0/3399286980 shutdown_connections 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.525+0000 7fec7d702640 1 -- 172.21.15.71:0/3399286980 >> 172.21.15.71:0/3399286980 conn(0x7fec78066a00 msgr2=0x7fec78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.525+0000 7fec7d702640 1 -- 172.21.15.71:0/3399286980 shutdown_connections 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.525+0000 7fec7d702640 1 -- 172.21.15.71:0/3399286980 wait complete. 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.526+0000 7fec7d702640 10 monclient: build_initial_monmap 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.526+0000 7fec7d702640 10 monclient: monmap: 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:17.529 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.526+0000 7fec7d702640 1 librados: starting msgr at 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.526+0000 7fec7d702640 1 librados: starting objecter 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.527+0000 7fec7d702640 1 Processor -- start 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.527+0000 7fec7d702640 1 -- start start 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.527+0000 7fec7d702640 1 librados: setting wanted keys 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.527+0000 7fec7d702640 1 librados: calling monclient init 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.527+0000 7fec7d702640 10 monclient: init 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:17.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:17.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 10 monclient(hunting): picked mon.noname-b con 0x7fec780a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 10 monclient(hunting): picked mon.noname-a con 0x7fec7813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 10 monclient(hunting): picked mon.noname-c con 0x7fec781457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7813a230 con 0x7fec7813f650 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec7813b2e0 con 0x7fec780a9d70 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec78138e60 con 0x7fec781457f0 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.528+0000 7fec7d702640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec7d702640 10 monclient(hunting): authenticate will time out at 1211.683053s 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec777fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fec781457f0 0x7fec781d7b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49978/0 (socket says 172.21.15.71:49978) 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec777fe640 1 -- 172.21.15.71:0/478136245 learned_addr learned my addr 172.21.15.71:0/478136245 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 10 client.?.objecter ms_handle_connect 0x7fec781457f0 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49601739 0 0) 0x7fec78138e60 con 0x7fec781457f0 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.529+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec64004eb0 con 0x7fec781457f0 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1235077752 0 0) 0x7fec64004eb0 con 0x7fec781457f0 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec78138e60 con 0x7fec781457f0 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fec680018e0 con 0x7fec781457f0 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 10 monclient(hunting): dump: 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:17.533 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:17.534 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:17.534 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:17.534 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:17.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.530+0000 7fec5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:17.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.531+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1946595254 0 0) 0x7fec78138e60 con 0x7fec781457f0 2024-03-29T11:39:17.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.531+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 >> v1:172.21.15.81:6789/0 conn(0x7fec780a9d70 legacy=0x7fec781398d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 >> v1:172.21.15.71:6789/0 conn(0x7fec7813f650 legacy=0x7fec781d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 1 monclient: found mon.c 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7813b9c0 con 0x7fec781457f0 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 client.?.objecter ms_handle_connect 0x7fec7813f650 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 10 monclient: _renew_subs 2024-03-29T11:39:17.537 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7813bbb0 con 0x7fec781457f0 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 10 monclient: _renew_subs 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec781e2f50 con 0x7fec781457f0 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec7d702640 1 librados: init done 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec68001d40 con 0x7fec781457f0 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fec68003a40 con 0x7fec781457f0 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.535+0000 7fec5f7fe640 10 monclient: dump: 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:17.538 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:17.539 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:17.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.537+0000 7fec757fa640 4 set_mon_vals no callback set 2024-03-29T11:39:17.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.537+0000 7fec7d702640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:17.540 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.537+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fec68003c20 con 0x7fec781457f0 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec7d702640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec7d702640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec58005080 con 0x7fec781457f0 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fec68084b30 con 0x7fec781457f0 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec5f7fe640 10 client.4529.objecter ms_dispatch 0x7fec78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec5f7fe640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec5f7fe640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec5f7fe640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.538+0000 7fec76ffd640 10 monclient: get_auth_request con 0x7fec6406db10 auth_method 0 2024-03-29T11:39:17.541 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.539+0000 7fec5f7fe640 10 client.4529.objecter ms_handle_connect 0x7fec6406db10 2024-03-29T11:39:17.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.547+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec68050410 con 0x7fec781457f0 2024-03-29T11:39:17.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.548+0000 7fec5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:17.549 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.548+0000 7fec5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.862+0000 7fec7d702640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.862+0000 7fec7d702640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.862+0000 7fec7d702640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.862+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec58005330 con 0x7fec781457f0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.863+0000 7fec5f7fe640 1 -- 172.21.15.71:0/478136245 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fec68052340 con 0x7fec781457f0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.863+0000 7fec5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.863+0000 7fec5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 10 librados: watch_flush enter 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 10 librados: watch_flush exit 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 10 client.4529.objecter successfully canceled tick 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 >> v1:172.21.15.71:6816/1596514235 conn(0x7fec6406db10 legacy=0x7fec6406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 10 monclient: shutdown 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.866+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 >> v1:172.21.15.71:6790/0 conn(0x7fec781457f0 legacy=0x7fec781d7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.867+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 shutdown_connections 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.867+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 >> 172.21.15.71:0/478136245 conn(0x7fec78066a00 msgr2=0x7fec780b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:17.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.869+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 shutdown_connections 2024-03-29T11:39:17.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.870+0000 7fec7d702640 1 -- 172.21.15.71:0/478136245 wait complete. 2024-03-29T11:39:17.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:17.872+0000 7fec7d702640 1 librados: shutdown 2024-03-29T11:39:17.892 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:24.698 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.847+0000 7efef202b640 10 monclient: get_monmap_and_config 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.847+0000 7efef202b640 10 monclient: build_initial_monmap 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.848+0000 7efef202b640 10 monclient: monmap: 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:24.848933+0000 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:24.848933+0000 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:24.849 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:24.850 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:24.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.849+0000 7efef202b640 1 Processor -- start 2024-03-29T11:39:24.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.849+0000 7efef202b640 1 -- start start 2024-03-29T11:39:24.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.849+0000 7efef202b640 10 monclient: init 2024-03-29T11:39:24.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.850+0000 7efef202b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:24.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.850+0000 7efef202b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:24.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.850+0000 7efef202b640 10 monclient(hunting): picked mon.noname-c con 0x7efee40389e0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 10 monclient(hunting): picked mon.noname-a con 0x7efee4038dc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef1029640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efee40389e0 0x7efee4033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:53616/0 (socket says 172.21.15.71:53616) 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef0828640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7efee4038dc0 0x7efee40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:46006/0 (socket says 172.21.15.71:46006) 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 10 monclient(hunting): picked mon.noname-b con 0x7efee4030660 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee40c54c0 con 0x7efee4038dc0 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef1029640 1 -- 172.21.15.71:0/3225368319 learned_addr learned my addr 172.21.15.71:0/3225368319 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee4036430 con 0x7efee40389e0 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee4039840 con 0x7efee4030660 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efef202b640 10 monclient(hunting): authenticate will time out at 1219.005442s 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609490812 0 0) 0x7efee4036430 con 0x7efee40389e0 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:24.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed8003520 con 0x7efee40389e0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370410117 0 0) 0x7efee40c54c0 con 0x7efee4038dc0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.851+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efee4036430 con 0x7efee4038dc0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2023819556 0 0) 0x7efee4036430 con 0x7efee4038dc0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 >> v1:172.21.15.71:6790/0 conn(0x7efee4030660 legacy=0x7efee40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 >> v1:172.21.15.81:6789/0 conn(0x7efee40389e0 legacy=0x7efee4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 1 monclient: found mon.noname-a 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee4035690 con 0x7efee4038dc0 2024-03-29T11:39:24.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efef202b640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efef202b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efedc001630 con 0x7efee4038dc0 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efee3fff640 10 monclient: dump: 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:24.858 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.852+0000 7efef202b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efedc001880 con 0x7efee4038dc0 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efee3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efef202b640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efef202b640 4 set_mon_vals no callback set 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efee3fff640 1 -- 172.21.15.71:0/3225368319 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efedc001c10 con 0x7efee4038dc0 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efee3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efee3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efee3fff640 10 monclient: dump: 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:24.859 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.853+0000 7efef202b640 10 monclient: shutdown 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.854+0000 7efef202b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.854+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 >> v1:172.21.15.71:6789/0 conn(0x7efee4038dc0 legacy=0x7efee40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.854+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 shutdown_connections 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.854+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 >> 172.21.15.71:0/3225368319 conn(0x7efee4005c00 msgr2=0x7efee4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.854+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 shutdown_connections 2024-03-29T11:39:24.860 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.854+0000 7efef202b640 1 -- 172.21.15.71:0/3225368319 wait complete. 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.855+0000 7efef202b640 10 monclient: build_initial_monmap 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.855+0000 7efef202b640 10 monclient: monmap: 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:24.864 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.855+0000 7efef202b640 1 librados: starting msgr at 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.855+0000 7efef202b640 1 librados: starting objecter 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.856+0000 7efef202b640 1 Processor -- start 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.856+0000 7efef202b640 1 -- start start 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.856+0000 7efef202b640 1 librados: setting wanted keys 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.856+0000 7efef202b640 1 librados: calling monclient init 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.856+0000 7efef202b640 10 monclient: init 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient(hunting): picked mon.noname-c con 0x7efee4030660 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:24.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient(hunting): picked mon.noname-b con 0x7efee40389e0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient(hunting): picked mon.noname-a con 0x7efee40c73f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee4030120 con 0x7efee40c73f0 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee415cf60 con 0x7efee40389e0 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee415cbd0 con 0x7efee4030660 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef202b640 10 monclient(hunting): authenticate will time out at 1219.011250s 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef1029640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7efee4030660 0x7efee415c4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53132/0 (socket says 172.21.15.71:53132) 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efef1029640 1 -- 172.21.15.71:0/1211157525 learned_addr learned my addr 172.21.15.71:0/1211157525 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 client.?.objecter ms_handle_connect 0x7efee40389e0 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 client.?.objecter ms_handle_connect 0x7efee4030660 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495943086 0 0) 0x7efee415cf60 con 0x7efee40389e0 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:24.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.857+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0004eb0 con 0x7efee40389e0 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4100248273 0 0) 0x7efed0004eb0 con 0x7efee40389e0 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efee415cf60 con 0x7efee40389e0 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efedc001bd0 con 0x7efee40389e0 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:24.867 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 client.?.objecter ms_handle_connect 0x7efee40c73f0 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587964051 0 0) 0x7efee415cbd0 con 0x7efee4030660 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efed0004eb0 con 0x7efee4030660 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 914152571 0 0) 0x7efee415cf60 con 0x7efee40389e0 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 >> v1:172.21.15.71:6790/0 conn(0x7efee4030660 legacy=0x7efee415c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 >> v1:172.21.15.71:6789/0 conn(0x7efee40c73f0 legacy=0x7efee4167050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:24.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 monclient: found mon.b 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee402d860 con 0x7efee40389e0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efef202b640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efef202b640 10 monclient: _renew_subs 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efef202b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.858+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee402db10 con 0x7efee40389e0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efef202b640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efef202b640 10 monclient: _renew_subs 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efef202b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efee4168950 con 0x7efee40389e0 2024-03-29T11:39:24.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efef202b640 1 librados: init done 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efedc004a90 con 0x7efee40389e0 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7efedc004ea0 con 0x7efee40389e0 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee1ffb640 10 monclient: dump: 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:24.870 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.859+0000 7efee37fe640 4 set_mon_vals no callback set 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.860+0000 7efef202b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.860+0000 7efef202b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.860+0000 7efef202b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.860+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efecc005080 con 0x7efee40389e0 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.861+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7efedc005080 con 0x7efee40389e0 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7efedc084c20 con 0x7efee40389e0 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efee1ffb640 10 client.4492.objecter ms_dispatch 0x7efeec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efee1ffb640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efee1ffb640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efee1ffb640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efef1029640 10 monclient: get_auth_request con 0x7efed006dd70 auth_method 0 2024-03-29T11:39:24.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.862+0000 7efee1ffb640 10 client.4492.objecter ms_handle_connect 0x7efed006dd70 2024-03-29T11:39:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.869+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efedc050560 con 0x7efee40389e0 2024-03-29T11:39:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.869+0000 7efee1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:24.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:24.869+0000 7efee1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:25.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.219+0000 7efef202b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:25.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.219+0000 7efef202b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:25.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.219+0000 7efef202b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:25.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.219+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efecc005330 con 0x7efee40389e0 2024-03-29T11:39:25.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.220+0000 7efee1ffb640 1 -- 172.21.15.71:0/1211157525 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7efedc050560 con 0x7efee40389e0 2024-03-29T11:39:25.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.220+0000 7efee1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:25.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.220+0000 7efee1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:25.222 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:25.222 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:25.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 10 librados: watch_flush enter 2024-03-29T11:39:25.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 10 librados: watch_flush exit 2024-03-29T11:39:25.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 10 client.4492.objecter successfully canceled tick 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 >> v1:172.21.15.71:6816/1596514235 conn(0x7efed006dd70 legacy=0x7efed0070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 10 monclient: shutdown 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.224+0000 7efef202b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.225+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 >> v1:172.21.15.81:6789/0 conn(0x7efee40389e0 legacy=0x7efee402d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.225+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 shutdown_connections 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.225+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 >> 172.21.15.71:0/1211157525 conn(0x7efee4005c00 msgr2=0x7efee4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.225+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 shutdown_connections 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.225+0000 7efef202b640 1 -- 172.21.15.71:0/1211157525 wait complete. 2024-03-29T11:39:25.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:25.225+0000 7efef202b640 1 librados: shutdown 2024-03-29T11:39:25.241 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:32.047 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:32.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.216+0000 7f85b3424640 10 monclient: get_monmap_and_config 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.216+0000 7f85b3424640 10 monclient: build_initial_monmap 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.216+0000 7f85b3424640 10 monclient: monmap: 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:32.217671+0000 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:32.217671+0000 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:32.219 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.217+0000 7f85b3424640 1 Processor -- start 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.217+0000 7f85b3424640 1 -- start start 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.217+0000 7f85b3424640 10 monclient: init 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient(hunting): picked mon.noname-b con 0x7f85ac0a9750 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient(hunting): picked mon.noname-a con 0x7f85ac0adca0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient(hunting): picked mon.noname-c con 0x7f85ac146a80 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0abdc0 con 0x7f85ac0adca0 2024-03-29T11:39:32.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0b0070 con 0x7f85ac146a80 2024-03-29T11:39:32.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0af880 con 0x7f85ac0a9750 2024-03-29T11:39:32.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:32.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.218+0000 7f85b3424640 10 monclient(hunting): authenticate will time out at 1226.372945s 2024-03-29T11:39:32.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.220+0000 7f85b2422640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f85ac0a9750 0x7f85ac143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39322/0 (socket says 172.21.15.71:39322) 2024-03-29T11:39:32.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.220+0000 7f85b2422640 1 -- 172.21.15.71:0/682146425 learned_addr learned my addr 172.21.15.71:0/682146425 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:32.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.221+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477737639 0 0) 0x7f85ac0af880 con 0x7f85ac0a9750 2024-03-29T11:39:32.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.221+0000 7f85b1420640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.221+0000 7f85b1420640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.221+0000 7f85b1420640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.221+0000 7f85b1420640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:39:32.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.221+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8598003520 con 0x7f85ac0a9750 2024-03-29T11:39:32.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.222+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4240021057 0 0) 0x7f8598003520 con 0x7f85ac0a9750 2024-03-29T11:39:32.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.222+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 >> v1:172.21.15.81:6789/0 conn(0x7f85ac146a80 legacy=0x7f85ac151bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:32.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.222+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 >> v1:172.21.15.71:6789/0 conn(0x7f85ac0adca0 legacy=0x7f85ac146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: _finish_hunting 0 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 1 monclient: found mon.noname-b 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0ac030 con 0x7f85ac0a9750 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: _finish_auth 0 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: _check_auth_tickets 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:32.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b3424640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b3424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f85a4001a10 con 0x7f85ac0a9750 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b1420640 10 monclient: dump: 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:32.227 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:32.228 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.225+0000 7f85b3424640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85a4001be0 con 0x7f85ac0a9750 2024-03-29T11:39:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b1420640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b3424640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:32.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b1420640 1 -- 172.21.15.71:0/682146425 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f85a4002030 con 0x7f85ac0a9750 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b1420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b3424640 4 set_mon_vals no callback set 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b1420640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.227+0000 7f85b1420640 10 monclient: dump: 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:32.229 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.228+0000 7f85b3424640 10 monclient: shutdown 2024-03-29T11:39:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.228+0000 7f85b3424640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:32.230 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.228+0000 7f85b3424640 1 -- 172.21.15.71:0/682146425 >> v1:172.21.15.71:6790/0 conn(0x7f85ac0a9750 legacy=0x7f85ac143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.228+0000 7f85b3424640 1 -- 172.21.15.71:0/682146425 shutdown_connections 2024-03-29T11:39:32.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.228+0000 7f85b3424640 1 -- 172.21.15.71:0/682146425 >> 172.21.15.71:0/682146425 conn(0x7f85ac0673e0 msgr2=0x7f85ac0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:32.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.229+0000 7f85b3424640 1 -- 172.21.15.71:0/682146425 shutdown_connections 2024-03-29T11:39:32.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.229+0000 7f85b3424640 1 -- 172.21.15.71:0/682146425 wait complete. 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 10 monclient: build_initial_monmap 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 10 monclient: monmap: 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 1 librados: starting msgr at 2024-03-29T11:39:32.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 1 librados: starting objecter 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 1 Processor -- start 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 1 -- start start 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 1 librados: setting wanted keys 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 1 librados: calling monclient init 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.230+0000 7f85b3424640 10 monclient: init 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient(hunting): picked mon.noname-c con 0x7f85ac0a9750 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:32.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient(hunting): picked mon.noname-b con 0x7f85ac0adca0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:32.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient(hunting): picked mon.noname-a con 0x7f85ac146a80 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:32.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac144880 con 0x7f85ac146a80 2024-03-29T11:39:32.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac13ebd0 con 0x7f85ac0adca0 2024-03-29T11:39:32.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac1db760 con 0x7f85ac0a9750 2024-03-29T11:39:32.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:32.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.231+0000 7f85b3424640 10 monclient(hunting): authenticate will time out at 1226.385807s 2024-03-29T11:39:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85b2422640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f85ac0a9750 0x7f85ac144170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39680/0 (socket says 172.21.15.71:39680) 2024-03-29T11:39:32.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85b2422640 1 -- 172.21.15.71:0/3056826761 learned_addr learned my addr 172.21.15.71:0/3056826761 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 client.?.objecter ms_handle_connect 0x7f85ac0adca0 2024-03-29T11:39:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 client.?.objecter ms_handle_connect 0x7f85ac146a80 2024-03-29T11:39:32.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 client.?.objecter ms_handle_connect 0x7f85ac0a9750 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155435622 0 0) 0x7f85ac13ebd0 con 0x7f85ac0adca0 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.235+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8590004eb0 con 0x7f85ac0adca0 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020862550 0 0) 0x7f85ac144880 con 0x7f85ac146a80 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:39:32.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ac13ebd0 con 0x7f85ac146a80 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446411701 0 0) 0x7f85ac1db760 con 0x7f85ac0a9750 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ac144880 con 0x7f85ac0a9750 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 541997724 0 0) 0x7f8590004eb0 con 0x7f85ac0adca0 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85ac1db760 con 0x7f85ac0adca0 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f859c0017f0 con 0x7f85ac0adca0 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:32.244 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 425719904 0 0) 0x7f85ac13ebd0 con 0x7f85ac146a80 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.236+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8590004eb0 con 0x7f85ac146a80 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f85a8001670 con 0x7f85ac146a80 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:32.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2021123643 0 0) 0x7f85ac1db760 con 0x7f85ac0adca0 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 >> v1:172.21.15.71:6790/0 conn(0x7f85ac0a9750 legacy=0x7f85ac144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 >> v1:172.21.15.71:6789/0 conn(0x7f85ac146a80 legacy=0x7f85ac13e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.246 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 monclient: found mon.b 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac1461f0 con 0x7f85ac0adca0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 10 monclient: _renew_subs 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac072560 con 0x7f85ac0adca0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 10 monclient: _renew_subs 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac1db360 con 0x7f85ac0adca0 2024-03-29T11:39:32.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85b3424640 1 librados: init done 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f859c001950 con 0x7f85ac0adca0 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.237+0000 7f85a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.238+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f859c003790 con 0x7f85ac0adca0 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.238+0000 7f85b0c1f640 4 set_mon_vals no callback set 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.238+0000 7f85a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.238+0000 7f85a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.238+0000 7f85a2ffd640 10 monclient: dump: 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:32.248 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.239+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f859c019760 con 0x7f85ac0adca0 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85b2422640 10 monclient: get_auth_request con 0x7f859006df60 auth_method 0 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f859c0836d0 con 0x7f85ac0adca0 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85a2ffd640 10 client.4495.objecter ms_dispatch 0x7f85ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85a2ffd640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85a2ffd640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85a2ffd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.240+0000 7f85b3424640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.241+0000 7f85b3424640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.241+0000 7f85b3424640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.241+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8594005080 con 0x7f85ac0adca0 2024-03-29T11:39:32.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.241+0000 7f85a2ffd640 10 client.4495.objecter ms_handle_connect 0x7f859006df60 2024-03-29T11:39:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.247+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f859c04ede0 con 0x7f85ac0adca0 2024-03-29T11:39:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.247+0000 7f85a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:32.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.247+0000 7f85a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:32.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.578+0000 7f85b3424640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:32.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.578+0000 7f85b3424640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:32.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.578+0000 7f85b3424640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:32.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.578+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8594005330 con 0x7f85ac0adca0 2024-03-29T11:39:32.584 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:32.584 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:32.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.581+0000 7f85a2ffd640 1 -- 172.21.15.71:0/3056826761 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f859c050ee0 con 0x7f85ac0adca0 2024-03-29T11:39:32.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.581+0000 7f85a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:32.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.581+0000 7f85a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:32.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 10 librados: watch_flush enter 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 10 librados: watch_flush exit 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 >> v1:172.21.15.71:6816/1596514235 conn(0x7f859006df60 legacy=0x7f8590070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 10 monclient: shutdown 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:32.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.585+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 >> v1:172.21.15.81:6789/0 conn(0x7f85ac0adca0 legacy=0x7f85ac13ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.587+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 shutdown_connections 2024-03-29T11:39:32.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.587+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 >> 172.21.15.71:0/3056826761 conn(0x7f85ac0673e0 msgr2=0x7f85ac0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.588+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 shutdown_connections 2024-03-29T11:39:32.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.591+0000 7f85b3424640 1 -- 172.21.15.71:0/3056826761 wait complete. 2024-03-29T11:39:32.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:32.591+0000 7f85b3424640 1 librados: shutdown 2024-03-29T11:39:32.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:39.419 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:39.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.552+0000 7f8fb380f640 10 monclient: get_monmap_and_config 2024-03-29T11:39:39.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.552+0000 7f8fb380f640 10 monclient: build_initial_monmap 2024-03-29T11:39:39.555 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.552+0000 7f8fb380f640 10 monclient: monmap: 2024-03-29T11:39:39.555 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:39.555 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:39.555 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:39.554809+0000 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:39.554809+0000 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:39.556 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.553+0000 7f8fb380f640 1 Processor -- start 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.553+0000 7f8fb380f640 1 -- start start 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.553+0000 7f8fb380f640 10 monclient: init 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient(hunting): picked mon.noname-b con 0x7f8fac0b4790 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient(hunting): picked mon.noname-c con 0x7f8fac0aea90 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:39.558 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient(hunting): picked mon.noname-a con 0x7f8fac0b39f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac0b0c60 con 0x7f8fac0b39f0 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac0af020 con 0x7f8fac0aea90 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac13fbd0 con 0x7f8fac0b4790 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.554+0000 7f8fb380f640 10 monclient(hunting): authenticate will time out at 1233.710031s 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fb15ab640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8fac0b4790 0x7f8fac0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35232/0 (socket says 172.21.15.71:35232) 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fb15ab640 1 -- 172.21.15.71:0/1895559202 learned_addr learned my addr 172.21.15.71:0/1895559202 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033102602 0 0) 0x7f8fac13fbd0 con 0x7f8fac0b4790 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:39:39.559 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f98003520 con 0x7f8fac0b4790 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110732477 0 0) 0x7f8fac0b0c60 con 0x7f8fac0b39f0 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f98003840 con 0x7f8fac0b39f0 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418109466 0 0) 0x7f8f98003520 con 0x7f8fac0b4790 2024-03-29T11:39:39.560 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.555+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 >> v1:172.21.15.81:6789/0 conn(0x7f8fac0aea90 legacy=0x7f8fac0af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 >> v1:172.21.15.71:6789/0 conn(0x7f8fac0b39f0 legacy=0x7f8fac13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 1 monclient: found mon.noname-b 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac13fd60 con 0x7f8fac0b4790 2024-03-29T11:39:39.561 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8fa8001670 con 0x7f8fac0b4790 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: dump: 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.562 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:39.563 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:39.563 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:39.563 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fb380f640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:39:39.563 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fb380f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fa8001850 con 0x7f8fac0b4790 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 1 -- 172.21.15.71:0/1895559202 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8fa8001c60 con 0x7f8fac0b4790 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fa3fff640 10 monclient: dump: 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:39.566 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fb380f640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.559+0000 7f8fb380f640 4 set_mon_vals no callback set 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.560+0000 7f8fb380f640 10 monclient: shutdown 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.560+0000 7f8fb380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.560+0000 7f8fb380f640 1 -- 172.21.15.71:0/1895559202 >> v1:172.21.15.71:6790/0 conn(0x7f8fac0b4790 legacy=0x7f8fac0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.560+0000 7f8fb380f640 1 -- 172.21.15.71:0/1895559202 shutdown_connections 2024-03-29T11:39:39.567 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.560+0000 7f8fb380f640 1 -- 172.21.15.71:0/1895559202 >> 172.21.15.71:0/1895559202 conn(0x7f8fac066a00 msgr2=0x7f8fac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.561+0000 7f8fb380f640 1 -- 172.21.15.71:0/1895559202 shutdown_connections 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.561+0000 7f8fb380f640 1 -- 172.21.15.71:0/1895559202 wait complete. 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 10 monclient: build_initial_monmap 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 10 monclient: monmap: 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.568 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 1 librados: starting msgr at 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 1 librados: starting objecter 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 1 Processor -- start 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 1 -- start start 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 1 librados: setting wanted keys 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 1 librados: calling monclient init 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.562+0000 7f8fb380f640 10 monclient: init 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.563+0000 7f8fb380f640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.563+0000 7f8fb380f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.563+0000 7f8fb380f640 10 monclient(hunting): picked mon.noname-c con 0x7f8fac0b39f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.563+0000 7f8fb15ab640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f8fac0b39f0 0x7f8fac1dec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:35618/0 (socket says 172.21.15.71:35618) 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.563+0000 7f8fb15ab640 1 -- 172.21.15.71:0/2122778065 learned_addr learned my addr 172.21.15.71:0/2122778065 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.564+0000 7f8fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f8fac0b39f0 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.564+0000 7f8fa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.564+0000 7f8fb380f640 10 monclient(hunting): picked mon.noname-a con 0x7f8fac0b4790 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:39.569 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.564+0000 7f8fa1ffb640 10 client.?.objecter ms_handle_connect 0x7f8fac0b4790 2024-03-29T11:39:39.570 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.564+0000 7f8fa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fb380f640 10 monclient(hunting): picked mon.noname-b con 0x7f8fac1420c0 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac142750 con 0x7f8fac0b4790 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac132640 con 0x7f8fac1420c0 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fac142be0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fb380f640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fb380f640 10 monclient(hunting): authenticate will time out at 1233.720457s 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281306891 0 0) 0x7f8fac142be0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.576 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f90004eb0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244544324 0 0) 0x7f8fac142750 con 0x7f8fac0b4790 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.565+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac142be0 con 0x7f8fac0b4790 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058604862 0 0) 0x7f8f90004eb0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fac142750 con 0x7f8fac0b39f0 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8fa8001c60 con 0x7f8fac0b39f0 2024-03-29T11:39:39.577 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:39.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659424064 0 0) 0x7f8fac142be0 con 0x7f8fac0b4790 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f90004eb0 con 0x7f8fac0b4790 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8f9c001870 con 0x7f8fac0b4790 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:39.579 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4135489481 0 0) 0x7f8fac142750 con 0x7f8fac0b39f0 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.566+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 >> v1:172.21.15.81:6789/0 conn(0x7f8fac1420c0 legacy=0x7f8fac131f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 >> v1:172.21.15.71:6789/0 conn(0x7f8fac0b4790 legacy=0x7f8fac131820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 1 monclient: found mon.c 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac134a10 con 0x7f8fac0b39f0 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fb380f640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fb380f640 10 monclient: _renew_subs 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fb380f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:39.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fac134c60 con 0x7f8fac0b39f0 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8fa8004a90 con 0x7f8fac0b39f0 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f8fa8004e20 con 0x7f8fac0b39f0 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa1ffb640 10 monclient: dump: 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:39.581 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.567+0000 7f8fa37fe640 4 set_mon_vals no callback set 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.569+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f8fa801adf0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.569+0000 7f8fb380f640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.569+0000 7f8fb380f640 10 monclient: _renew_subs 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.569+0000 7f8fb380f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.569+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fac1df530 con 0x7f8fac0b39f0 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.569+0000 7f8fb380f640 1 librados: init done 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.570+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f8fa8084d40 con 0x7f8fac0b39f0 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.570+0000 7f8fa1ffb640 10 client.4544.objecter ms_dispatch 0x7f8fac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.570+0000 7f8fa1ffb640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.570+0000 7f8fa1ffb640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.570+0000 7f8fb0daa640 10 monclient: get_auth_request con 0x7f8f9006ddb0 auth_method 0 2024-03-29T11:39:39.582 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.570+0000 7f8fa1ffb640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.571+0000 7f8fa1ffb640 10 client.4544.objecter ms_handle_connect 0x7f8f9006ddb0 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.571+0000 7f8fb380f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.571+0000 7f8fb380f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.571+0000 7f8fb380f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.571+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fac1df7e0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.578+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8fa8050680 con 0x7f8fac0b39f0 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.578+0000 7f8fa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:39.583 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.578+0000 7f8fa1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:39.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.886+0000 7f8fb380f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:39.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.886+0000 7f8fb380f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:39.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.886+0000 7f8fb380f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:39.888 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.886+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fac075f10 con 0x7f8fac0b39f0 2024-03-29T11:39:39.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.887+0000 7f8fa1ffb640 1 -- 172.21.15.71:0/2122778065 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f8fa80843f0 con 0x7f8fac0b39f0 2024-03-29T11:39:39.889 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.887+0000 7f8fa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:39.890 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.887+0000 7f8fa1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:39.890 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:39.890 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.889+0000 7f8fb380f640 10 librados: watch_flush enter 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.889+0000 7f8fb380f640 10 librados: watch_flush exit 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.889+0000 7f8fb380f640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.889+0000 7f8fb380f640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.890+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 >> v1:172.21.15.71:6816/1596514235 conn(0x7f8f9006ddb0 legacy=0x7f8f90070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.890+0000 7f8fb380f640 10 monclient: shutdown 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.890+0000 7f8fb380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:39.892 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.890+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 >> v1:172.21.15.71:6790/0 conn(0x7f8fac0b39f0 legacy=0x7f8fac1dec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.890+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 shutdown_connections 2024-03-29T11:39:39.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.890+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 >> 172.21.15.71:0/2122778065 conn(0x7f8fac066a00 msgr2=0x7f8fac0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:39.893 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.891+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 shutdown_connections 2024-03-29T11:39:39.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.891+0000 7f8fb380f640 1 -- 172.21.15.71:0/2122778065 wait complete. 2024-03-29T11:39:39.894 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:39.891+0000 7f8fb380f640 1 librados: shutdown 2024-03-29T11:39:39.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:46.715 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:46.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.847+0000 7f3ddabf6640 10 monclient: get_monmap_and_config 2024-03-29T11:39:46.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.847+0000 7f3ddabf6640 10 monclient: build_initial_monmap 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.847+0000 7f3ddabf6640 10 monclient: monmap: 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:46.848331+0000 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:46.848331+0000 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:46.849 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.848+0000 7f3ddabf6640 1 Processor -- start 2024-03-29T11:39:46.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.848+0000 7f3ddabf6640 1 -- start start 2024-03-29T11:39:46.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.848+0000 7f3ddabf6640 10 monclient: init 2024-03-29T11:39:46.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.849+0000 7f3ddabf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:46.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.849+0000 7f3ddabf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:46.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.849+0000 7f3ddabf6640 10 monclient(hunting): picked mon.noname-b con 0x7f3dd40b4710 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:46.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.849+0000 7f3ddabf6640 10 monclient(hunting): picked mon.noname-c con 0x7f3dd40b3970 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.849+0000 7f3ddabf6640 10 monclient(hunting): picked mon.noname-a con 0x7f3dd413f390 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.849+0000 7f3dd3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3dd40b3970 0x7f3dd40af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:42408/0 (socket says 172.21.15.71:42408) 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd3fff640 1 -- 172.21.15.71:0/2461617824 learned_addr learned my addr 172.21.15.71:0/2461617824 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3ddabf6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40af020 con 0x7f3dd413f390 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40b0910 con 0x7f3dd40b3970 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd8992640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3dd40b4710 0x7f3dd40b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36780/0 (socket says 172.21.15.71:36780) 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40b0bf0 con 0x7f3dd40b4710 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3ddabf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3ddabf6640 10 monclient(hunting): authenticate will time out at 1241.004529s 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957355812 0 0) 0x7f3dd40b0bf0 con 0x7f3dd40b4710 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:46.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:46.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd37fe640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:39:46.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.850+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc4003520 con 0x7f3dd40b4710 2024-03-29T11:39:46.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.851+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1272269358 0 0) 0x7f3dc4003520 con 0x7f3dd40b4710 2024-03-29T11:39:46.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.851+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 >> v1:172.21.15.81:6789/0 conn(0x7f3dd40b3970 legacy=0x7f3dd40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:46.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.851+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 >> v1:172.21.15.71:6789/0 conn(0x7f3dd413f390 legacy=0x7f3dd413f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.852+0000 7f3dd37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 1 monclient: found mon.noname-b 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd40aeb00 con 0x7f3dd40b4710 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3dc8001690 con 0x7f3dd40b4710 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:46.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: dump: 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3ddabf6640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3ddabf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dc8001860 con 0x7f3dd40b4710 2024-03-29T11:39:46.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 1 -- 172.21.15.71:0/2461617824 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3dc8001cb0 con 0x7f3dd40b4710 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3dd37fe640 10 monclient: dump: 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:46.858 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3ddabf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.853+0000 7f3ddabf6640 4 set_mon_vals no callback set 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 10 monclient: shutdown 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 >> v1:172.21.15.71:6790/0 conn(0x7f3dd40b4710 legacy=0x7f3dd40b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 shutdown_connections 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 >> 172.21.15.71:0/2461617824 conn(0x7f3dd4066a00 msgr2=0x7f3dd4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 shutdown_connections 2024-03-29T11:39:46.859 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.854+0000 7f3ddabf6640 1 -- 172.21.15.71:0/2461617824 wait complete. 2024-03-29T11:39:46.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.855+0000 7f3ddabf6640 10 monclient: build_initial_monmap 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.855+0000 7f3ddabf6640 10 monclient: monmap: 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 1 librados: starting msgr at 2024-03-29T11:39:46.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 1 librados: starting objecter 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 1 Processor -- start 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 1 -- start start 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 1 librados: setting wanted keys 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 1 librados: calling monclient init 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 10 monclient: init 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.856+0000 7f3ddabf6640 10 monclient(hunting): picked mon.noname-a con 0x7f3dd40b3970 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 10 monclient(hunting): picked mon.noname-b con 0x7f3dd40b4710 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 10 monclient(hunting): picked mon.noname-c con 0x7f3dd413f390 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd41da920 con 0x7f3dd40b3970 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd41da580 con 0x7f3dd40b4710 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd41394f0 con 0x7f3dd413f390 2024-03-29T11:39:46.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3ddabf6640 10 monclient(hunting): authenticate will time out at 1241.011315s 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3dd9193640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f3dd413f390 0x7f3dd41e2920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:37870/0 (socket says 172.21.15.71:37870) 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3dd9193640 1 -- 172.21.15.71:0/1482787569 learned_addr learned my addr 172.21.15.71:0/1482787569 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3dd17fa640 10 client.?.objecter ms_handle_connect 0x7f3dd413f390 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3dd17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.857+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765003862 0 0) 0x7f3dd41394f0 con 0x7f3dd413f390 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db8004eb0 con 0x7f3dd413f390 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 client.?.objecter ms_handle_connect 0x7f3dd40b4710 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664295083 0 0) 0x7f3dd41da580 con 0x7f3dd40b4710 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:46.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd41394f0 con 0x7f3dd40b4710 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3673946525 0 0) 0x7f3dd41394f0 con 0x7f3dd40b4710 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dd41da580 con 0x7f3dd40b4710 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3dc0001510 con 0x7f3dd40b4710 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): dump: 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.869 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1337346336 0 0) 0x7f3db8004eb0 con 0x7f3dd413f390 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3dd41394f0 con 0x7f3dd413f390 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.858+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3dcc001ad0 con 0x7f3dd413f390 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.859+0000 7f3dd17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.859+0000 7f3dd17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.859+0000 7f3dd17fa640 10 monclient(hunting): dump: 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.870 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.859+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2564002541 0 0) 0x7f3dd41da580 con 0x7f3dd40b4710 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.859+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 >> v1:172.21.15.71:6790/0 conn(0x7f3dd413f390 legacy=0x7f3dd41e2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.859+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 >> v1:172.21.15.71:6789/0 conn(0x7f3dd40b3970 legacy=0x7f3dd41dae40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 1 monclient: found mon.b 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4139680 con 0x7f3dd40b4710 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:46.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3dd17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 10 monclient: _renew_subs 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd4138f30 con 0x7f3dd40b4710 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 10 monclient: _renew_subs 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd4133600 con 0x7f3dd40b4710 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.860+0000 7f3ddabf6640 1 librados: init done 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dc0001a40 con 0x7f3dd40b4710 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f3dc0004b10 con 0x7f3dd40b4710 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd17fa640 10 monclient: dump: 2024-03-29T11:39:46.872 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.861+0000 7f3dd2ffd640 4 set_mon_vals no callback set 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.862+0000 7f3ddabf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.862+0000 7f3ddabf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.862+0000 7f3ddabf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.862+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db4005080 con 0x7f3dd40b4710 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.863+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f3dc001aae0 con 0x7f3dd40b4710 2024-03-29T11:39:46.873 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f3dc00849e0 con 0x7f3dd40b4710 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd17fa640 10 client.4498.objecter ms_dispatch 0x7f3dd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd17fa640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd17fa640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd17fa640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd8992640 10 monclient: get_auth_request con 0x7f3db806de40 auth_method 0 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.864+0000 7f3dd17fa640 10 client.4498.objecter ms_handle_connect 0x7f3db806de40 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.871+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3dc00502c0 con 0x7f3dd40b4710 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.871+0000 7f3dd17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:46.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:46.871+0000 7f3dd17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.174+0000 7f3ddabf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.174+0000 7f3ddabf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.174+0000 7f3ddabf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.174+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3db4005330 con 0x7f3dd40b4710 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.175+0000 7f3dd17fa640 1 -- 172.21.15.71:0/1482787569 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f3dc00521f0 con 0x7f3dd40b4710 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.175+0000 7f3dd17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.175+0000 7f3dd17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:47.179 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:47.180 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:47.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 10 librados: watch_flush enter 2024-03-29T11:39:47.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 10 librados: watch_flush exit 2024-03-29T11:39:47.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-29T11:39:47.180 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 10 client.4498.objecter successfully canceled tick 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 >> v1:172.21.15.71:6816/1596514235 conn(0x7f3db806de40 legacy=0x7f3db8070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 10 monclient: shutdown 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.179+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 >> v1:172.21.15.81:6789/0 conn(0x7f3dd40b4710 legacy=0x7f3dd4132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.180+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 shutdown_connections 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.180+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 >> 172.21.15.71:0/1482787569 conn(0x7f3dd4066a00 msgr2=0x7f3dd40b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.181+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 shutdown_connections 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.181+0000 7f3ddabf6640 1 -- 172.21.15.71:0/1482787569 wait complete. 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:47.181+0000 7f3ddabf6640 1 librados: shutdown 2024-03-29T11:39:47.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:54.002 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:54.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.134+0000 7f28261b5640 10 monclient: get_monmap_and_config 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.134+0000 7f28261b5640 10 monclient: build_initial_monmap 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.134+0000 7f28261b5640 10 monclient: monmap: 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:39:54.135134+0000 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:39:54.135134+0000 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:39:54.136 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:54.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.135+0000 7f28261b5640 1 Processor -- start 2024-03-29T11:39:54.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.135+0000 7f28261b5640 1 -- start start 2024-03-29T11:39:54.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.135+0000 7f28261b5640 10 monclient: init 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.136+0000 7f28261b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.136+0000 7f28261b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.136+0000 7f28261b5640 10 monclient(hunting): picked mon.noname-b con 0x7f28200b1f60 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.136+0000 7f28261b5640 10 monclient(hunting): picked mon.noname-a con 0x7f2820071eb0 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f28200b1f60 0x7f2820143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47400/0 (socket says 172.21.15.71:47400) 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f28261b5640 10 monclient(hunting): picked mon.noname-c con 0x7f2820143d10 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281f7fe640 1 -- 172.21.15.71:0/858100700 learned_addr learned my addr 172.21.15.71:0/858100700 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282006ef80 con 0x7f2820071eb0 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282006f260 con 0x7f2820143d10 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28200b40c0 con 0x7f28200b1f60 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f28261b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f28261b5640 10 monclient(hunting): authenticate will time out at 1248.291235s 2024-03-29T11:39:54.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480104373 0 0) 0x7f28200b40c0 con 0x7f28200b1f60 2024-03-29T11:39:54.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:54.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:54.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:54.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:39:54.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c003520 con 0x7f28200b1f60 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103240796 0 0) 0x7f282006f260 con 0x7f2820143d10 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28200b40c0 con 0x7f2820143d10 2024-03-29T11:39:54.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436504080 0 0) 0x7f280c003520 con 0x7f28200b1f60 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 >> v1:172.21.15.81:6789/0 conn(0x7f2820143d10 legacy=0x7f2820146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.137+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 >> v1:172.21.15.71:6789/0 conn(0x7f2820071eb0 legacy=0x7f2820146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 1 monclient: found mon.noname-b 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2820155d70 con 0x7f28200b1f60 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: _finish_auth 0 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2810001690 con 0x7f28200b1f60 2024-03-29T11:39:54.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f281effd640 10 monclient: dump: 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f28261b5640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.138+0000 7f28261b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28100017f0 con 0x7f28200b1f60 2024-03-29T11:39:54.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f281effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f281effd640 1 -- 172.21.15.71:0/858100700 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2810001c00 con 0x7f28200b1f60 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f281effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f281effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f281effd640 10 monclient: dump: 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:54.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f28261b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f28261b5640 4 set_mon_vals no callback set 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f28261b5640 10 monclient: shutdown 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f28261b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.139+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 >> v1:172.21.15.71:6790/0 conn(0x7f28200b1f60 legacy=0x7f2820143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.140+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 shutdown_connections 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.140+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 >> 172.21.15.71:0/858100700 conn(0x7f2820066a00 msgr2=0x7f2820066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.140+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 shutdown_connections 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.140+0000 7f28261b5640 1 -- 172.21.15.71:0/858100700 wait complete. 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.141+0000 7f28261b5640 10 monclient: build_initial_monmap 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.141+0000 7f28261b5640 10 monclient: monmap: 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:39:54.146 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:54.147 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:54.147 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:39:54.147 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:39:54.147 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:39:54.147 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.141+0000 7f28261b5640 1 librados: starting msgr at 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.141+0000 7f28261b5640 1 librados: starting objecter 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.141+0000 7f28261b5640 1 Processor -- start 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 1 -- start start 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 1 librados: setting wanted keys 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 1 librados: calling monclient init 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient: init 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient(hunting): picked mon.noname-c con 0x7f28200b1f60 addr v1:172.21.15.71:6790/0 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient(hunting): picked mon.noname-b con 0x7f2820143d10 addr v1:172.21.15.81:6789/0 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient(hunting): picked mon.noname-a con 0x7f2820149340 addr v1:172.21.15.71:6789/0 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282013fd40 con 0x7f2820149340 2024-03-29T11:39:54.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282014af90 con 0x7f2820143d10 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282014b200 con 0x7f28200b1f60 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f28261b5640 10 monclient(hunting): authenticate will time out at 1248.296699s 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f281f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f28200b1f60 0x7f282013f630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47532/0 (socket says 172.21.15.71:47532) 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f281f7fe640 1 -- 172.21.15.71:0/2407880953 learned_addr learned my addr 172.21.15.71:0/2407880953 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f281cff9640 10 client.?.objecter ms_handle_connect 0x7f28200b1f60 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.142+0000 7f281cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.143+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015426810 0 0) 0x7f282014b200 con 0x7f28200b1f60 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.143+0000 7f281cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.143+0000 7f281cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.143+0000 7f281cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.143+0000 7f281cff9640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.143+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2800004eb0 con 0x7f28200b1f60 2024-03-29T11:39:54.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 428039766 0 0) 0x7f2800004eb0 con 0x7f28200b1f60 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2800005300 con 0x7f28200b1f60 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2810001c00 con 0x7f28200b1f60 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 10 monclient(hunting): dump: 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:54.154 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1163959660 0 0) 0x7f2800005300 con 0x7f28200b1f60 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 >> v1:172.21.15.81:6789/0 conn(0x7f2820143d10 legacy=0x7f282013b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.144+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 >> v1:172.21.15.71:6789/0 conn(0x7f2820149340 legacy=0x7f28201e6f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 1 monclient: found mon.c 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282014b4e0 con 0x7f28200b1f60 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 10 monclient: _finish_auth 0 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f281cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 10 monclient: _renew_subs 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28201499c0 con 0x7f28200b1f60 2024-03-29T11:39:54.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 10 monclient: _renew_subs 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2820070ae0 con 0x7f28200b1f60 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.146+0000 7f28261b5640 1 librados: init done 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2810004b10 con 0x7f28200b1f60 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2810001c00 con 0x7f28200b1f60 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281cff9640 10 monclient: dump: 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.156 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.147+0000 7f281e7fc640 4 set_mon_vals no callback set 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.148+0000 7f28261b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.148+0000 7f28261b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.148+0000 7f28261b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.148+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2804005080 con 0x7f28200b1f60 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.149+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f2810004e40 con 0x7f28200b1f60 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f2816ffd640 10 monclient: get_auth_request con 0x7f280006e130 auth_method 0 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f28100847a0 con 0x7f28200b1f60 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f281cff9640 10 client.4556.objecter ms_dispatch 0x7f2820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f281cff9640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f281cff9640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:54.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f281cff9640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:39:54.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.150+0000 7f281cff9640 10 client.4556.objecter ms_handle_connect 0x7f280006e130 2024-03-29T11:39:54.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.157+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28100500e0 con 0x7f28200b1f60 2024-03-29T11:39:54.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.157+0000 7f281cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:54.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.157+0000 7f281cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:54.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.458+0000 7f28261b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:54.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.458+0000 7f28261b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:54.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.458+0000 7f28261b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:39:54.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.458+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2804005330 con 0x7f28200b1f60 2024-03-29T11:39:54.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.459+0000 7f281cff9640 1 -- 172.21.15.71:0/2407880953 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f2810001a60 con 0x7f28200b1f60 2024-03-29T11:39:54.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.459+0000 7f281cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:54.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.459+0000 7f281cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:54.461 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:39:54.461 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:54.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 10 librados: watch_flush enter 2024-03-29T11:39:54.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 10 librados: watch_flush exit 2024-03-29T11:39:54.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:39:54.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:39:54.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 >> v1:172.21.15.71:6816/1596514235 conn(0x7f280006e130 legacy=0x7f28000705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 10 monclient: shutdown 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 >> v1:172.21.15.71:6790/0 conn(0x7f28200b1f60 legacy=0x7f282013f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 shutdown_connections 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 >> 172.21.15.71:0/2407880953 conn(0x7f2820066a00 msgr2=0x7f28200b7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 shutdown_connections 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 -- 172.21.15.71:0/2407880953 wait complete. 2024-03-29T11:39:54.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:39:54.462+0000 7f28261b5640 1 librados: shutdown 2024-03-29T11:39:54.478 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:01.284 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:01.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.431+0000 7f24e7495640 10 monclient: get_monmap_and_config 2024-03-29T11:40:01.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.431+0000 7f24e7495640 10 monclient: build_initial_monmap 2024-03-29T11:40:01.434 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.431+0000 7f24e7495640 10 monclient: monmap: 2024-03-29T11:40:01.434 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:01.434 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:01.434 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:01.432666+0000 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:01.432666+0000 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.432+0000 7f24e7495640 1 Processor -- start 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.432+0000 7f24e7495640 1 -- start start 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.432+0000 7f24e7495640 10 monclient: init 2024-03-29T11:40:01.435 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient(hunting): picked mon.noname-c con 0x7f24e00a9750 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient(hunting): picked mon.noname-b con 0x7f24e0149ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient(hunting): picked mon.noname-a con 0x7f24e014fe50 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e00aa4a0 con 0x7f24e014fe50 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e014a680 con 0x7f24e00a9750 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e00af290 con 0x7f24e0149ac0 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.433+0000 7f24e7495640 10 monclient(hunting): authenticate will time out at 1255.587873s 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.434+0000 7f24e4a30640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f24e0149ac0 0x7f24e0149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56604/0 (socket says 172.21.15.71:56604) 2024-03-29T11:40:01.436 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.434+0000 7f24e4a30640 1 -- 172.21.15.71:0/59252298 learned_addr learned my addr 172.21.15.71:0/59252298 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:01.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.436+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90827827 0 0) 0x7f24e00af290 con 0x7f24e0149ac0 2024-03-29T11:40:01.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.436+0000 7f24d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:01.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.436+0000 7f24d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:01.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.436+0000 7f24d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:01.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.436+0000 7f24d7fff640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:40:01.438 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.436+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24cc003520 con 0x7f24e0149ac0 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154078521 0 0) 0x7f24cc003520 con 0x7f24e0149ac0 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 >> v1:172.21.15.81:6789/0 conn(0x7f24e00a9750 legacy=0x7f24e01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 >> v1:172.21.15.71:6789/0 conn(0x7f24e014fe50 legacy=0x7f24e0152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 monclient: found mon.noname-b 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:01.442 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00afb40 con 0x7f24e0149ac0 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f24dc001690 con 0x7f24e0149ac0 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: dump: 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.443 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc001920 con 0x7f24e0149ac0 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 1 -- 172.21.15.71:0/59252298 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f24dc001cb0 con 0x7f24e0149ac0 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.439+0000 7f24d7fff640 10 monclient: dump: 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.444 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.440+0000 7f24e7495640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.440+0000 7f24e7495640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.440+0000 7f24e7495640 4 set_mon_vals no callback set 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.441+0000 7f24e7495640 10 monclient: shutdown 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.441+0000 7f24e7495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:01.445 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.441+0000 7f24e7495640 1 -- 172.21.15.71:0/59252298 >> v1:172.21.15.71:6790/0 conn(0x7f24e0149ac0 legacy=0x7f24e0149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.442+0000 7f24e7495640 1 -- 172.21.15.71:0/59252298 shutdown_connections 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.442+0000 7f24e7495640 1 -- 172.21.15.71:0/59252298 >> 172.21.15.71:0/59252298 conn(0x7f24e00b35e0 msgr2=0x7f24e00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.442+0000 7f24e7495640 1 -- 172.21.15.71:0/59252298 shutdown_connections 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.442+0000 7f24e7495640 1 -- 172.21.15.71:0/59252298 wait complete. 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 10 monclient: build_initial_monmap 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 10 monclient: monmap: 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:01.447 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 1 librados: starting msgr at 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 1 librados: starting objecter 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 1 Processor -- start 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 1 -- start start 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 1 librados: setting wanted keys 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 1 librados: calling monclient init 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.443+0000 7f24e7495640 10 monclient: init 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 10 monclient(hunting): picked mon.noname-a con 0x7f24e00a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:01.448 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 10 monclient(hunting): picked mon.noname-c con 0x7f24e0149ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:01.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 10 monclient(hunting): picked mon.noname-b con 0x7f24e014fe50 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:01.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e0148f50 con 0x7f24e00a9750 2024-03-29T11:40:01.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e0149230 con 0x7f24e014fe50 2024-03-29T11:40:01.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e013ff10 con 0x7f24e0149ac0 2024-03-29T11:40:01.449 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.444+0000 7f24e7495640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:01.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.445+0000 7f24e4a30640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f24e0149ac0 0x7f24e013e400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:57906/0 (socket says 172.21.15.71:57906) 2024-03-29T11:40:01.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.445+0000 7f24e4a30640 1 -- 172.21.15.71:0/2025204895 learned_addr learned my addr 172.21.15.71:0/2025204895 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:01.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.445+0000 7f24d5ffb640 10 client.?.objecter ms_handle_connect 0x7f24e0149ac0 2024-03-29T11:40:01.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.445+0000 7f24d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:01.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.445+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33675712 0 0) 0x7f24e013ff10 con 0x7f24e0149ac0 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24e7495640 10 monclient(hunting): authenticate will time out at 1255.602474s 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24d5ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24c4004eb0 con 0x7f24e0149ac0 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.448+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 775917780 0 0) 0x7f24c4004eb0 con 0x7f24e0149ac0 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24e013ff10 con 0x7f24e0149ac0 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f24dc001610 con 0x7f24e0149ac0 2024-03-29T11:40:01.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:01.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2734145117 0 0) 0x7f24e013ff10 con 0x7f24e0149ac0 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 >> v1:172.21.15.81:6789/0 conn(0x7f24e014fe50 legacy=0x7f24e01ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 >> v1:172.21.15.71:6789/0 conn(0x7f24e00a9750 legacy=0x7f24e01e44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 monclient: found mon.c 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e013dd90 con 0x7f24e0149ac0 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 10 monclient: _renew_subs 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:01.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e013b010 con 0x7f24e0149ac0 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 10 monclient: _renew_subs 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e0140e70 con 0x7f24e0149ac0 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.449+0000 7f24e7495640 1 librados: init done 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc004c60 con 0x7f24e0149ac0 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f24dc004ff0 con 0x7f24e0149ac0 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d5ffb640 10 monclient: dump: 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.457 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.451+0000 7f24d77fe640 4 set_mon_vals no callback set 2024-03-29T11:40:01.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.452+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f24dc005210 con 0x7f24e0149ac0 2024-03-29T11:40:01.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.452+0000 7f24e7495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:01.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24e7495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:01.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24e7495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c0005080 con 0x7f24e0149ac0 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f24dc084cd0 con 0x7f24e0149ac0 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24d5ffb640 10 client.4562.objecter ms_dispatch 0x7f24e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24d5ffb640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24d5ffb640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24e5231640 10 monclient: get_auth_request con 0x7f24c406db80 auth_method 0 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.453+0000 7f24d5ffb640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:40:01.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.454+0000 7f24d5ffb640 10 client.4562.objecter ms_handle_connect 0x7f24c406db80 2024-03-29T11:40:01.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.464+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24dc050610 con 0x7f24e0149ac0 2024-03-29T11:40:01.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.464+0000 7f24d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:01.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.464+0000 7f24d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.784+0000 7f24e7495640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.784+0000 7f24e7495640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.784+0000 7f24e7495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.784+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24c0005330 con 0x7f24e0149ac0 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.785+0000 7f24d5ffb640 1 -- 172.21.15.71:0/2025204895 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f24dc050610 con 0x7f24e0149ac0 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.785+0000 7f24d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.785+0000 7f24d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:01.788 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:01.788 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:01.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.789+0000 7f24e7495640 10 librados: watch_flush enter 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 10 librados: watch_flush exit 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 >> v1:172.21.15.71:6816/1596514235 conn(0x7f24c406db80 legacy=0x7f24c4070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 10 monclient: shutdown 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 >> v1:172.21.15.71:6790/0 conn(0x7f24e0149ac0 legacy=0x7f24e013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 shutdown_connections 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.792+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 >> 172.21.15.71:0/2025204895 conn(0x7f24e00b35e0 msgr2=0x7f24e00b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.793+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 shutdown_connections 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.793+0000 7f24e7495640 1 -- 172.21.15.71:0/2025204895 wait complete. 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:01.793+0000 7f24e7495640 1 librados: shutdown 2024-03-29T11:40:01.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:08.617 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:08.763 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.761+0000 7fef2f34d640 10 monclient: get_monmap_and_config 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.761+0000 7fef2f34d640 10 monclient: build_initial_monmap 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.761+0000 7fef2f34d640 10 monclient: monmap: 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:08.762477+0000 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:08.762477+0000 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:08.764 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:08.765 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.762+0000 7fef2f34d640 1 Processor -- start 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.762+0000 7fef2f34d640 1 -- start start 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.762+0000 7fef2f34d640 10 monclient: init 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient(hunting): picked mon.noname-c con 0x7fef2814a400 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient(hunting): picked mon.noname-b con 0x7fef280aa500 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient(hunting): picked mon.noname-a con 0x7fef28149800 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:08.766 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280aca50 con 0x7fef28149800 2024-03-29T11:40:08.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef28149cb0 con 0x7fef2814a400 2024-03-29T11:40:08.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280acc80 con 0x7fef280aa500 2024-03-29T11:40:08.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:08.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.763+0000 7fef2f34d640 10 monclient(hunting): authenticate will time out at 1262.917859s 2024-03-29T11:40:08.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.765+0000 7fef2db4a640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fef280aa500 0x7fef280ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39352/0 (socket says 172.21.15.71:39352) 2024-03-29T11:40:08.767 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.765+0000 7fef2db4a640 1 -- 172.21.15.71:0/2250019332 learned_addr learned my addr 172.21.15.71:0/2250019332 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419890640 0 0) 0x7fef28149cb0 con 0x7fef2814a400 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): _negotiate 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef14003520 con 0x7fef2814a400 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437556519 0 0) 0x7fef280aca50 con 0x7fef28149800 2024-03-29T11:40:08.773 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): _negotiate 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.768+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef28149cb0 con 0x7fef28149800 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3282381658 0 0) 0x7fef28149cb0 con 0x7fef28149800 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 >> v1:172.21.15.71:6790/0 conn(0x7fef280aa500 legacy=0x7fef280ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 >> v1:172.21.15.81:6789/0 conn(0x7fef2814a400 legacy=0x7fef280aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: _finish_hunting 0 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 1 monclient: found mon.noname-a 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280ac350 con 0x7fef28149800 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: _finish_auth 0 2024-03-29T11:40:08.774 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: _check_auth_tickets 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fef200019f0 con 0x7fef28149800 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.769+0000 7fef2d349640 10 monclient: dump: 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:08.775 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.770+0000 7fef2f34d640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.770+0000 7fef2f34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.770+0000 7fef2d349640 1 -- 172.21.15.71:0/2250019332 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef20001b50 con 0x7fef28149800 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.770+0000 7fef2d349640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.770+0000 7fef2f34d640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.770+0000 7fef2f34d640 4 set_mon_vals no callback set 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.771+0000 7fef2f34d640 10 monclient: shutdown 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.771+0000 7fef2f34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.771+0000 7fef2f34d640 1 -- 172.21.15.71:0/2250019332 >> v1:172.21.15.71:6789/0 conn(0x7fef28149800 legacy=0x7fef28151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.772+0000 7fef2f34d640 1 -- 172.21.15.71:0/2250019332 shutdown_connections 2024-03-29T11:40:08.776 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.772+0000 7fef2f34d640 1 -- 172.21.15.71:0/2250019332 >> 172.21.15.71:0/2250019332 conn(0x7fef280b3a30 msgr2=0x7fef280b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.773+0000 7fef2f34d640 1 -- 172.21.15.71:0/2250019332 shutdown_connections 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.773+0000 7fef2f34d640 1 -- 172.21.15.71:0/2250019332 wait complete. 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.774+0000 7fef2f34d640 10 monclient: build_initial_monmap 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.774+0000 7fef2f34d640 10 monclient: monmap: 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:08.777 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.774+0000 7fef2f34d640 1 librados: starting msgr at 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.774+0000 7fef2f34d640 1 librados: starting objecter 2024-03-29T11:40:08.778 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.775+0000 7fef2f34d640 1 Processor -- start 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.775+0000 7fef2f34d640 1 -- start start 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.775+0000 7fef2f34d640 1 librados: setting wanted keys 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.775+0000 7fef2f34d640 1 librados: calling monclient init 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.775+0000 7fef2f34d640 10 monclient: init 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.775+0000 7fef2f34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 10 monclient(hunting): picked mon.noname-b con 0x7fef280aa500 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 10 monclient(hunting): picked mon.noname-a con 0x7fef28149800 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:08.779 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 10 monclient(hunting): picked mon.noname-c con 0x7fef2814a400 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2814c090 con 0x7fef28149800 2024-03-29T11:40:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2806f560 con 0x7fef280aa500 2024-03-29T11:40:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2814c2e0 con 0x7fef2814a400 2024-03-29T11:40:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:08.780 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.776+0000 7fef2f34d640 10 monclient(hunting): authenticate will time out at 1262.930132s 2024-03-29T11:40:08.781 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.778+0000 7fef2eb4c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fef2814a400 0x7fef2806f9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:44332/0 (socket says 172.21.15.71:44332) 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.778+0000 7fef2eb4c640 1 -- 172.21.15.71:0/1599575142 learned_addr learned my addr 172.21.15.71:0/1599575142 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.779+0000 7fef1effd640 10 client.?.objecter ms_handle_connect 0x7fef28149800 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.779+0000 7fef1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025237599 0 0) 0x7fef2814c090 con 0x7fef28149800 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0c004eb0 con 0x7fef28149800 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 client.?.objecter ms_handle_connect 0x7fef2814a400 2024-03-29T11:40:08.782 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:08.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 134684215 0 0) 0x7fef0c004eb0 con 0x7fef28149800 2024-03-29T11:40:08.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef2814c090 con 0x7fef28149800 2024-03-29T11:40:08.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fef180018a0 con 0x7fef28149800 2024-03-29T11:40:08.785 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): dump: 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:08.786 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175199886 0 0) 0x7fef2814c2e0 con 0x7fef2814a400 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.780+0000 7fef1effd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0c004eb0 con 0x7fef2814a400 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1338905447 0 0) 0x7fef0c004eb0 con 0x7fef2814a400 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef2814c2e0 con 0x7fef2814a400 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fef20003b10 con 0x7fef2814a400 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient(hunting): dump: 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:08.787 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 24903622 0 0) 0x7fef2814c090 con 0x7fef28149800 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 >> v1:172.21.15.71:6790/0 conn(0x7fef2814a400 legacy=0x7fef2806f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 >> v1:172.21.15.81:6789/0 conn(0x7fef280aa500 legacy=0x7fef280721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 monclient: found mon.a 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:40:08.788 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2814acc0 con 0x7fef28149800 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient: _finish_auth 0 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.781+0000 7fef1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 10 monclient: _renew_subs 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2814af10 con 0x7fef28149800 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 10 monclient: _renew_subs 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef28140e40 con 0x7fef28149800 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef2f34d640 1 librados: init done 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef180047e0 con 0x7fef28149800 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fef18004bf0 con 0x7fef28149800 2024-03-29T11:40:08.789 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef1effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.782+0000 7fef1effd640 10 monclient: dump: 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:08.790 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.783+0000 7fef1ffff640 4 set_mon_vals no callback set 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.785+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fef18004d50 con 0x7fef28149800 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.785+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fef180849f0 con 0x7fef28149800 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.785+0000 7fef1effd640 10 client.4482.objecter ms_dispatch 0x7fef28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.785+0000 7fef1effd640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.785+0000 7fef1effd640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.785+0000 7fef1effd640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.786+0000 7fef2f34d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.786+0000 7fef2f34d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:08.792 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.786+0000 7fef2f34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:40:08.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.786+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef10005080 con 0x7fef28149800 2024-03-29T11:40:08.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.786+0000 7fef2e34b640 10 monclient: get_auth_request con 0x7fef0c06df70 auth_method 0 2024-03-29T11:40:08.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.786+0000 7fef1effd640 10 client.4482.objecter ms_handle_connect 0x7fef0c06df70 2024-03-29T11:40:08.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.796+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef18050340 con 0x7fef28149800 2024-03-29T11:40:08.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.796+0000 7fef1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:08.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:08.796+0000 7fef1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.130+0000 7fef2f34d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.130+0000 7fef2f34d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.130+0000 7fef2f34d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.130+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef10005330 con 0x7fef28149800 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.131+0000 7fef1effd640 1 -- 172.21.15.71:0/1599575142 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fef18052270 con 0x7fef28149800 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.131+0000 7fef1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.131+0000 7fef1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 10 librados: watch_flush enter 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 10 librados: watch_flush exit 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 >> v1:172.21.15.71:6816/1596514235 conn(0x7fef0c06df70 legacy=0x7fef0c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 10 monclient: shutdown 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 >> v1:172.21.15.71:6789/0 conn(0x7fef28149800 legacy=0x7fef28140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 shutdown_connections 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.135+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 >> 172.21.15.71:0/1599575142 conn(0x7fef280b3a30 msgr2=0x7fef280b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.136+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 shutdown_connections 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.136+0000 7fef2f34d640 1 -- 172.21.15.71:0/1599575142 wait complete. 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:09.136+0000 7fef2f34d640 1 librados: shutdown 2024-03-29T11:40:09.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:15.961 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:16.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.106+0000 7fd4d95b5640 10 monclient: get_monmap_and_config 2024-03-29T11:40:16.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.106+0000 7fd4d95b5640 10 monclient: build_initial_monmap 2024-03-29T11:40:16.108 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.106+0000 7fd4d95b5640 10 monclient: monmap: 2024-03-29T11:40:16.108 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:16.107615+0000 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:16.107615+0000 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:16.109 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.108+0000 7fd4d95b5640 1 Processor -- start 2024-03-29T11:40:16.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.108+0000 7fd4d95b5640 1 -- start start 2024-03-29T11:40:16.110 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.108+0000 7fd4d95b5640 10 monclient: init 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd4d41457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d40a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d40aaef0 con 0x7fd4d413f650 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4145bd0 con 0x7fd4d41457f0 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4145eb0 con 0x7fd4d40a9d70 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.109+0000 7fd4d95b5640 10 monclient(hunting): authenticate will time out at 1270.263183s 2024-03-29T11:40:16.111 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.110+0000 7fd4d37fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd4d40a9d70 0x7fd4d414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:60708/0 (socket says 172.21.15.71:60708) 2024-03-29T11:40:16.112 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.110+0000 7fd4d37fe640 1 -- 172.21.15.71:0/3851895615 learned_addr learned my addr 172.21.15.71:0/3851895615 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.113+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 783646528 0 0) 0x7fd4d4145eb0 con 0x7fd4d40a9d70 2024-03-29T11:40:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:40:16.115 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4bc003520 con 0x7fd4d40a9d70 2024-03-29T11:40:16.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3064921729 0 0) 0x7fd4bc003520 con 0x7fd4d40a9d70 2024-03-29T11:40:16.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 >> v1:172.21.15.81:6789/0 conn(0x7fd4d41457f0 legacy=0x7fd4d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.116 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.114+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 >> v1:172.21.15.71:6789/0 conn(0x7fd4d413f650 legacy=0x7fd4d40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:16.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:16.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:40:16.117 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4146340 con 0x7fd4d40a9d70 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd4c8001690 con 0x7fd4d40a9d70 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d1ffb640 10 monclient: dump: 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.118 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d95b5640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:40:16.119 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.116+0000 7fd4d95b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4c8001860 con 0x7fd4d40a9d70 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d1ffb640 1 -- 172.21.15.71:0/3851895615 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd4c8001cb0 con 0x7fd4d40a9d70 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d1ffb640 10 monclient: dump: 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:16.120 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:16.121 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:16.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d95b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:16.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.117+0000 7fd4d95b5640 4 set_mon_vals no callback set 2024-03-29T11:40:16.121 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.118+0000 7fd4d95b5640 10 monclient: shutdown 2024-03-29T11:40:16.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.118+0000 7fd4d95b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:16.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.118+0000 7fd4d95b5640 1 -- 172.21.15.71:0/3851895615 >> v1:172.21.15.71:6790/0 conn(0x7fd4d40a9d70 legacy=0x7fd4d414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.119+0000 7fd4d95b5640 1 -- 172.21.15.71:0/3851895615 shutdown_connections 2024-03-29T11:40:16.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.119+0000 7fd4d95b5640 1 -- 172.21.15.71:0/3851895615 >> 172.21.15.71:0/3851895615 conn(0x7fd4d4066a00 msgr2=0x7fd4d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:16.122 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.119+0000 7fd4d95b5640 1 -- 172.21.15.71:0/3851895615 shutdown_connections 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.120+0000 7fd4d95b5640 1 -- 172.21.15.71:0/3851895615 wait complete. 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.120+0000 7fd4d95b5640 10 monclient: build_initial_monmap 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.120+0000 7fd4d95b5640 10 monclient: monmap: 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:16.123 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.121+0000 7fd4d95b5640 1 librados: starting msgr at 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.121+0000 7fd4d95b5640 1 librados: starting objecter 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.121+0000 7fd4d95b5640 1 Processor -- start 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.123+0000 7fd4d95b5640 1 -- start start 2024-03-29T11:40:16.124 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.123+0000 7fd4d95b5640 1 librados: setting wanted keys 2024-03-29T11:40:16.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.123+0000 7fd4d95b5640 1 librados: calling monclient init 2024-03-29T11:40:16.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.123+0000 7fd4d95b5640 10 monclient: init 2024-03-29T11:40:16.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:16.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient(hunting): picked mon.noname-b con 0x7fd4d40a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient(hunting): picked mon.noname-a con 0x7fd4d413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient(hunting): picked mon.noname-c con 0x7fd4d41457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4142d20 con 0x7fd4d413f650 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4143060 con 0x7fd4d40a9d70 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4135b80 con 0x7fd4d41457f0 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:16.126 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.124+0000 7fd4d95b5640 10 monclient(hunting): authenticate will time out at 1270.278449s 2024-03-29T11:40:16.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.125+0000 7fd4d37fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fd4d41457f0 0x7fd4d41e1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48582/0 (socket says 172.21.15.71:48582) 2024-03-29T11:40:16.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.125+0000 7fd4d37fe640 1 -- 172.21.15.71:0/1551556980 learned_addr learned my addr 172.21.15.71:0/1551556980 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:16.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.127+0000 7fd4bb7fe640 10 client.?.objecter ms_handle_connect 0x7fd4d41457f0 2024-03-29T11:40:16.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.127+0000 7fd4bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152144931 0 0) 0x7fd4d4135b80 con 0x7fd4d41457f0 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c0004eb0 con 0x7fd4d41457f0 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 10 client.?.objecter ms_handle_connect 0x7fd4d413f650 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.128+0000 7fd4bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1449531111 0 0) 0x7fd4c0004eb0 con 0x7fd4d41457f0 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4c0005300 con 0x7fd4d41457f0 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd4c8001b70 con 0x7fd4d41457f0 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:16.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176633833 0 0) 0x7fd4d4142d20 con 0x7fd4d413f650 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:40:16.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.129+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c0004eb0 con 0x7fd4d413f650 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.130+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1220841734 0 0) 0x7fd4c0005300 con 0x7fd4d41457f0 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.130+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 >> v1:172.21.15.81:6789/0 conn(0x7fd4d40a9d70 legacy=0x7fd4d4140010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 >> v1:172.21.15.71:6789/0 conn(0x7fd4d413f650 legacy=0x7fd4d4140bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 1 monclient: found mon.c 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4135d10 con 0x7fd4d41457f0 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4c8004bf0 con 0x7fd4d41457f0 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:16.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fd4c8004f00 con 0x7fd4d41457f0 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4bb7fe640 10 monclient: dump: 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 10 monclient: _renew_subs 2024-03-29T11:40:16.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d4142840 con 0x7fd4d41457f0 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 10 monclient: _renew_subs 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d41414b0 con 0x7fd4d41457f0 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.132+0000 7fd4d95b5640 1 librados: init done 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.133+0000 7fd4d17fa640 4 set_mon_vals no callback set 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.134+0000 7fd4b97fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.134+0000 7fd4b97fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.134+0000 7fd4b97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:16.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.134+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d4142b90 con 0x7fd4d41457f0 2024-03-29T11:40:16.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.138+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fd4c801aed0 con 0x7fd4d41457f0 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.138+0000 7fd4d2ffd640 10 monclient: get_auth_request con 0x7fd4c006dcb0 auth_method 0 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.138+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fd4c8054870 con 0x7fd4d41457f0 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.138+0000 7fd4bb7fe640 10 client.4574.objecter ms_dispatch 0x7fd4d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.139+0000 7fd4bb7fe640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.139+0000 7fd4bb7fe640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.139+0000 7fd4bb7fe640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:40:16.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.139+0000 7fd4bb7fe640 10 client.4574.objecter ms_handle_connect 0x7fd4c006dcb0 2024-03-29T11:40:16.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.145+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4c8050b10 con 0x7fd4d41457f0 2024-03-29T11:40:16.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.145+0000 7fd4bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:16.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.145+0000 7fd4bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:16.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.472+0000 7fd4b97fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:16.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.472+0000 7fd4b97fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:16.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.472+0000 7fd4b97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:16.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.472+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4d4142550 con 0x7fd4d41457f0 2024-03-29T11:40:16.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.473+0000 7fd4bb7fe640 1 -- 172.21.15.71:0/1551556980 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fd4c8050cb0 con 0x7fd4d41457f0 2024-03-29T11:40:16.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.473+0000 7fd4bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:16.474 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.473+0000 7fd4bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:16.476 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:16.476 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.476+0000 7fd4b97fa640 10 librados: watch_flush enter 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.476+0000 7fd4b97fa640 10 librados: watch_flush exit 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.476+0000 7fd4b97fa640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 >> v1:172.21.15.71:6816/1596514235 conn(0x7fd4c006dcb0 legacy=0x7fd4c0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 10 monclient: shutdown 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 >> v1:172.21.15.71:6790/0 conn(0x7fd4d41457f0 legacy=0x7fd4d41e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 shutdown_connections 2024-03-29T11:40:16.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 >> 172.21.15.71:0/1551556980 conn(0x7fd4d4066a00 msgr2=0x7fd4d413b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:16.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.477+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 shutdown_connections 2024-03-29T11:40:16.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.478+0000 7fd4b97fa640 1 -- 172.21.15.71:0/1551556980 wait complete. 2024-03-29T11:40:16.480 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:16.478+0000 7fd4b97fa640 1 librados: shutdown 2024-03-29T11:40:16.496 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:23.302 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:23.453 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.451+0000 7f98cfed9640 10 monclient: get_monmap_and_config 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.451+0000 7f98cfed9640 10 monclient: build_initial_monmap 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.451+0000 7f98cfed9640 10 monclient: monmap: 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:23.452634+0000 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:23.452634+0000 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:23.454 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.454+0000 7f98cfed9640 1 Processor -- start 2024-03-29T11:40:23.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.454+0000 7f98cfed9640 1 -- start start 2024-03-29T11:40:23.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.454+0000 7f98cfed9640 10 monclient: init 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient(hunting): picked mon.noname-c con 0x7f98c81422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient(hunting): picked mon.noname-b con 0x7f98c8142690 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient(hunting): picked mon.noname-a con 0x7f98c80b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c8142c80 con 0x7f98c80b17d0 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c80aae40 con 0x7f98c81422b0 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c813fbd0 con 0x7f98c8142690 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:23.457 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.455+0000 7f98cfed9640 10 monclient(hunting): authenticate will time out at 1277.609463s 2024-03-29T11:40:23.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.458+0000 7f98cd474640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f98c8142690 0x7f98c80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51051/0 (socket says 172.21.15.71:51051) 2024-03-29T11:40:23.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.458+0000 7f98cd474640 1 -- 172.21.15.71:0/2226385566 learned_addr learned my addr 172.21.15.71:0/2226385566 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196307207 0 0) 0x7f98c813fbd0 con 0x7f98c8142690 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b4003520 con 0x7f98c8142690 2024-03-29T11:40:23.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.461+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395472373 0 0) 0x7f98c8142c80 con 0x7f98c80b17d0 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c813fbd0 con 0x7f98c80b17d0 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736838505 0 0) 0x7f98c80aae40 con 0x7f98c81422b0 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c8142c80 con 0x7f98c81422b0 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 397428096 0 0) 0x7f98b4003520 con 0x7f98c8142690 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 >> v1:172.21.15.81:6789/0 conn(0x7f98c81422b0 legacy=0x7f98c813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 >> v1:172.21.15.71:6789/0 conn(0x7f98c80b17d0 legacy=0x7f98c80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: _finish_hunting 0 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 monclient: found mon.noname-b 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8151940 con 0x7f98c8142690 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: _finish_auth 0 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: _check_auth_tickets 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98cfed9640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98cfed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f98c4001690 con 0x7f98c8142690 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:23.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: dump: 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c40018b0 con 0x7f98c8142690 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 1 -- 172.21.15.71:0/2226385566 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f98c4001bc0 con 0x7f98c8142690 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:23.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98ccc73640 10 monclient: dump: 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98cfed9640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:23.468 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.462+0000 7f98cfed9640 4 set_mon_vals no callback set 2024-03-29T11:40:23.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.463+0000 7f98cfed9640 10 monclient: shutdown 2024-03-29T11:40:23.469 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.463+0000 7f98cfed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.463+0000 7f98cfed9640 1 -- 172.21.15.71:0/2226385566 >> v1:172.21.15.71:6790/0 conn(0x7f98c8142690 legacy=0x7f98c80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.464+0000 7f98cfed9640 1 -- 172.21.15.71:0/2226385566 shutdown_connections 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.464+0000 7f98cfed9640 1 -- 172.21.15.71:0/2226385566 >> 172.21.15.71:0/2226385566 conn(0x7f98c8066a00 msgr2=0x7f98c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.466+0000 7f98cfed9640 1 -- 172.21.15.71:0/2226385566 shutdown_connections 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.466+0000 7f98cfed9640 1 -- 172.21.15.71:0/2226385566 wait complete. 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.468+0000 7f98cfed9640 10 monclient: build_initial_monmap 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.468+0000 7f98cfed9640 10 monclient: monmap: 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.470 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:23.471 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:23.471 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:23.471 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.468+0000 7f98cfed9640 1 librados: starting msgr at 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.468+0000 7f98cfed9640 1 librados: starting objecter 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.469+0000 7f98cfed9640 1 Processor -- start 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.469+0000 7f98cfed9640 1 -- start start 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.469+0000 7f98cfed9640 1 librados: setting wanted keys 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.469+0000 7f98cfed9640 1 librados: calling monclient init 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.469+0000 7f98cfed9640 10 monclient: init 2024-03-29T11:40:23.472 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.470+0000 7f98cfed9640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.470+0000 7f98cfed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 10 monclient(hunting): picked mon.noname-c con 0x7f98c80b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 10 monclient(hunting): picked mon.noname-b con 0x7f98c81422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 10 monclient(hunting): picked mon.noname-a con 0x7f98c8142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c80af9b0 con 0x7f98c8142690 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c80afb60 con 0x7f98c81422b0 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c80b10d0 con 0x7f98c80b17d0 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:23.473 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.471+0000 7f98cfed9640 10 monclient(hunting): authenticate will time out at 1277.625497s 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98ce476640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f98c8142690 0x7f98c80af2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:45184/0 (socket says 172.21.15.71:45184) 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98ce476640 1 -- 172.21.15.71:0/3050801568 learned_addr learned my addr 172.21.15.71:0/3050801568 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 client.?.objecter ms_handle_connect 0x7f98c8142690 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 client.?.objecter ms_handle_connect 0x7f98c80b17d0 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:23.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756927817 0 0) 0x7f98c80b10d0 con 0x7f98c80b17d0 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98ac004eb0 con 0x7f98c80b17d0 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439078428 0 0) 0x7f98c80af9b0 con 0x7f98c8142690 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.472+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c80b10d0 con 0x7f98c8142690 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1229777230 0 0) 0x7f98c80b10d0 con 0x7f98c8142690 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c80af9b0 con 0x7f98c8142690 2024-03-29T11:40:23.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f98c0001c20 con 0x7f98c8142690 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 522934149 0 0) 0x7f98ac004eb0 con 0x7f98c80b17d0 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c80b10d0 con 0x7f98c80b17d0 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f98b8001820 con 0x7f98c80b17d0 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.473+0000 7f98be7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:23.478 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:23.479 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:23.479 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.474+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4196202389 0 0) 0x7f98c80b10d0 con 0x7f98c80b17d0 2024-03-29T11:40:23.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.474+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 >> v1:172.21.15.81:6789/0 conn(0x7f98c81422b0 legacy=0x7f98c80afff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:23.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 >> v1:172.21.15.71:6789/0 conn(0x7f98c8142690 legacy=0x7f98c80af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:23.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 1 monclient: found mon.c 2024-03-29T11:40:23.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8143500 con 0x7f98c80b17d0 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 10 monclient: _renew_subs 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c814daf0 con 0x7f98c80b17d0 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 10 monclient: _renew_subs 2024-03-29T11:40:23.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.476+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c81e32d0 con 0x7f98c80b17d0 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98cfed9640 1 librados: init done 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98b8001d30 con 0x7f98c80b17d0 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f98b80037b0 con 0x7f98c80b17d0 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.477+0000 7f98be7fc640 10 monclient: dump: 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:23.485 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.478+0000 7f98bf7fe640 4 set_mon_vals no callback set 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.478+0000 7f98cfed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.479+0000 7f98cfed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.479+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f98b8019780 con 0x7f98c80b17d0 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.479+0000 7f98cfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.479+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b0005080 con 0x7f98c80b17d0 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f98b8083810 con 0x7f98c80b17d0 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98be7fc640 10 client.4580.objecter ms_dispatch 0x7f98c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98be7fc640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98be7fc640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:23.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98be7fc640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:40:23.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98cd474640 10 monclient: get_auth_request con 0x7f98ac06de30 auth_method 0 2024-03-29T11:40:23.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.480+0000 7f98be7fc640 10 client.4580.objecter ms_handle_connect 0x7f98ac06de30 2024-03-29T11:40:23.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.488+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98b804ef20 con 0x7f98c80b17d0 2024-03-29T11:40:23.491 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.488+0000 7f98be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:23.492 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.488+0000 7f98be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.813+0000 7f98cfed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.813+0000 7f98cfed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.813+0000 7f98cfed9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.813+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98b0005330 con 0x7f98c80b17d0 2024-03-29T11:40:23.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.814+0000 7f98be7fc640 1 -- 172.21.15.71:0/3050801568 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f98b8051020 con 0x7f98c80b17d0 2024-03-29T11:40:23.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.814+0000 7f98be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:23.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.814+0000 7f98be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:23.816 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:23.817 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:23.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.818+0000 7f98cfed9640 10 librados: watch_flush enter 2024-03-29T11:40:23.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.818+0000 7f98cfed9640 10 librados: watch_flush exit 2024-03-29T11:40:23.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.818+0000 7f98cfed9640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:40:23.820 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.818+0000 7f98cfed9640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.820+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 >> v1:172.21.15.71:6816/1596514235 conn(0x7f98ac06de30 legacy=0x7f98ac0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.820+0000 7f98cfed9640 10 monclient: shutdown 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.820+0000 7f98cfed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.820+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 >> v1:172.21.15.71:6790/0 conn(0x7f98c80b17d0 legacy=0x7f98c80ac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.820+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 shutdown_connections 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.820+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 >> 172.21.15.71:0/3050801568 conn(0x7f98c8066a00 msgr2=0x7f98c813d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.821+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 shutdown_connections 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.822+0000 7f98cfed9640 1 -- 172.21.15.71:0/3050801568 wait complete. 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:23.822+0000 7f98cfed9640 1 librados: shutdown 2024-03-29T11:40:23.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:30.647 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.791+0000 7f554df89640 10 monclient: get_monmap_and_config 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.791+0000 7f554df89640 10 monclient: build_initial_monmap 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.791+0000 7f554df89640 10 monclient: monmap: 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:30.792161+0000 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:30.792161+0000 2024-03-29T11:40:30.793 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:30.794 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.794 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:30.794 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:30.794 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:30.794 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.792+0000 7f554df89640 1 Processor -- start 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.792+0000 7f554df89640 1 -- start start 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.792+0000 7f554df89640 10 monclient: init 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient(hunting): picked mon.noname-b con 0x7f55481457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:30.796 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient(hunting): picked mon.noname-c con 0x7f554813f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient(hunting): picked mon.noname-a con 0x7f55480a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f5547fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f55480a9d70 0x7f554814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:41832/0 (socket says 172.21.15.71:41832) 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f5547fff640 1 -- 172.21.15.71:0/3832288685 learned_addr learned my addr 172.21.15.71:0/3832288685 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55480aaef0 con 0x7f55480a9d70 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5548145bd0 con 0x7f554813f650 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5548145eb0 con 0x7f55481457f0 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.793+0000 7f554df89640 10 monclient(hunting): authenticate will time out at 1284.947997s 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997833786 0 0) 0x7f55480aaef0 con 0x7f55480a9d70 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:30.797 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:40:30.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5530003520 con 0x7f55480a9d70 2024-03-29T11:40:30.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125855450 0 0) 0x7f5530003520 con 0x7f55480a9d70 2024-03-29T11:40:30.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 >> v1:172.21.15.71:6790/0 conn(0x7f55481457f0 legacy=0x7f554813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:30.798 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.794+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 >> v1:172.21.15.81:6789/0 conn(0x7f554813f650 legacy=0x7f55480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:30.800 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 1 monclient: found mon.noname-a 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548146340 con 0x7f55480a9d70 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5540001a10 con 0x7f55480a9d70 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: dump: 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:30.801 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f554df89640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f554df89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5540001be0 con 0x7f55480a9d70 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.795+0000 7f55467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.796+0000 7f55467fc640 1 -- 172.21.15.71:0/3832288685 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5540002030 con 0x7f55480a9d70 2024-03-29T11:40:30.802 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.796+0000 7f55467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.796+0000 7f55467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.796+0000 7f55467fc640 10 monclient: dump: 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.796+0000 7f554df89640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.796+0000 7f554df89640 4 set_mon_vals no callback set 2024-03-29T11:40:30.803 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 10 monclient: shutdown 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 >> v1:172.21.15.71:6789/0 conn(0x7f55480a9d70 legacy=0x7f554814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 shutdown_connections 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 >> 172.21.15.71:0/3832288685 conn(0x7f5548066a00 msgr2=0x7f5548066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 shutdown_connections 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.797+0000 7f554df89640 1 -- 172.21.15.71:0/3832288685 wait complete. 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.798+0000 7f554df89640 10 monclient: build_initial_monmap 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.798+0000 7f554df89640 10 monclient: monmap: 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:30.804 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.798+0000 7f554df89640 1 librados: starting msgr at 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.798+0000 7f554df89640 1 librados: starting objecter 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.798+0000 7f554df89640 1 Processor -- start 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 1 -- start start 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 1 librados: setting wanted keys 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 1 librados: calling monclient init 2024-03-29T11:40:30.805 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 10 monclient: init 2024-03-29T11:40:30.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:30.809 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 10 monclient(hunting): picked mon.noname-b con 0x7f55480a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 10 monclient(hunting): picked mon.noname-a con 0x7f554813f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 10 monclient(hunting): picked mon.noname-c con 0x7f55481457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55481429b0 con 0x7f554813f650 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.799+0000 7f554df89640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5548137990 con 0x7f55480a9d70 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.800+0000 7f554df89640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5548142e40 con 0x7f55481457f0 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.800+0000 7f554df89640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.800+0000 7f554df89640 10 monclient(hunting): authenticate will time out at 1284.954141s 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.800+0000 7f5547fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f55481457f0 0x7f5548137280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56506/0 (socket says 172.21.15.71:56506) 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.800+0000 7f5547fff640 1 -- 172.21.15.71:0/611105009 learned_addr learned my addr 172.21.15.71:0/611105009 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 client.?.objecter ms_handle_connect 0x7f55481457f0 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:30.810 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186120029 0 0) 0x7f5548142e40 con 0x7f55481457f0 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5534004eb0 con 0x7f55481457f0 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 client.?.objecter ms_handle_connect 0x7f554813f650 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209001413 0 0) 0x7f5534004eb0 con 0x7f55481457f0 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5548142e40 con 0x7f55481457f0 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5540001ef0 con 0x7f55481457f0 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): dump: 2024-03-29T11:40:30.811 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.801+0000 7f552ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753010817 0 0) 0x7f55481429b0 con 0x7f554813f650 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:30.812 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5534004eb0 con 0x7f554813f650 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082517481 0 0) 0x7f5534004eb0 con 0x7f554813f650 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55481429b0 con 0x7f554813f650 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f553c0017d0 con 0x7f554813f650 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 10 monclient(hunting): dump: 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.813 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1952290794 0 0) 0x7f5548142e40 con 0x7f55481457f0 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.802+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 >> v1:172.21.15.81:6789/0 conn(0x7f55480a9d70 legacy=0x7f55481d74e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.803+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 >> v1:172.21.15.71:6789/0 conn(0x7f554813f650 legacy=0x7f5548136b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.803+0000 7f552ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.803+0000 7f552ffff640 1 monclient: found mon.c 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.803+0000 7f552ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.803+0000 7f552ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.803+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55480ac710 con 0x7f55481457f0 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 10 monclient: _finish_auth 0 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:30.814 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 10 monclient: _renew_subs 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55480ac9f0 con 0x7f55481457f0 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5540004f70 con 0x7f55481457f0 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5540005280 con 0x7f55481457f0 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f552ffff640 10 monclient: dump: 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:30.815 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 10 monclient: _renew_subs 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f5545ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55481d7bf0 con 0x7f55481457f0 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.804+0000 7f554df89640 1 librados: init done 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.806+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f554001b250 con 0x7f55481457f0 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.806+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f5540084f60 con 0x7f55481457f0 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f552ffff640 10 client.4583.objecter ms_dispatch 0x7f5548000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f552ffff640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:30.816 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f552ffff640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f552ffff640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f55477fe640 10 monclient: get_auth_request con 0x7f553406ddd0 auth_method 0 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f554df89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f554df89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f554df89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5548142b60 con 0x7f55481457f0 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.807+0000 7f552ffff640 10 client.4583.objecter ms_handle_connect 0x7f553406ddd0 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.815+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5548142b60 con 0x7f55481457f0 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.815+0000 7f552ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:30.817 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:30.815+0000 7f552ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.137+0000 7f554df89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.137+0000 7f554df89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.137+0000 7f554df89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.137+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5548142b60 con 0x7f55481457f0 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.138+0000 7f552ffff640 1 -- 172.21.15.71:0/611105009 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f5548142b60 con 0x7f55481457f0 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.138+0000 7f552ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:31.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.138+0000 7f552ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:31.141 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:31.141 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:31.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.141+0000 7f554df89640 10 librados: watch_flush enter 2024-03-29T11:40:31.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.141+0000 7f554df89640 10 librados: watch_flush exit 2024-03-29T11:40:31.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.141+0000 7f554df89640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:40:31.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.141+0000 7f554df89640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:40:31.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 >> v1:172.21.15.71:6816/1596514235 conn(0x7f553406ddd0 legacy=0x7f5534070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 10 monclient: shutdown 2024-03-29T11:40:31.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:31.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 >> v1:172.21.15.71:6790/0 conn(0x7f55481457f0 legacy=0x7f5548137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 shutdown_connections 2024-03-29T11:40:31.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 >> 172.21.15.71:0/611105009 conn(0x7f5548066a00 msgr2=0x7f55480b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:31.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 shutdown_connections 2024-03-29T11:40:31.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 -- 172.21.15.71:0/611105009 wait complete. 2024-03-29T11:40:31.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:31.142+0000 7f554df89640 1 librados: shutdown 2024-03-29T11:40:31.156 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:37.964 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.122+0000 7fe726298640 10 monclient: get_monmap_and_config 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.122+0000 7fe726298640 10 monclient: build_initial_monmap 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.122+0000 7fe726298640 10 monclient: monmap: 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:38.123996+0000 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:38.123996+0000 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:38.125 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:38.126 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:38.126 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:38.126 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:38.126 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:38.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.123+0000 7fe726298640 1 Processor -- start 2024-03-29T11:40:38.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.123+0000 7fe726298640 1 -- start start 2024-03-29T11:40:38.128 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.123+0000 7fe726298640 10 monclient: init 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient(hunting): picked mon.noname-a con 0x7fe7200b0860 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient(hunting): picked mon.noname-c con 0x7fe72013ef30 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient(hunting): picked mon.noname-b con 0x7fe7200af8e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200b0c40 con 0x7fe7200b0860 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200aed10 con 0x7fe72013ef30 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200afd50 con 0x7fe7200af8e0 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:38.129 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.124+0000 7fe726298640 10 monclient(hunting): authenticate will time out at 1292.279770s 2024-03-29T11:40:38.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.126+0000 7fe724835640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe7200af8e0 0x7fe720149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:39508/0 (socket says 172.21.15.71:39508) 2024-03-29T11:40:38.130 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.126+0000 7fe724835640 1 -- 172.21.15.71:0/1012491453 learned_addr learned my addr 172.21.15.71:0/1012491453 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:38.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.127+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934496930 0 0) 0x7fe7200afd50 con 0x7fe7200af8e0 2024-03-29T11:40:38.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.127+0000 7fe71effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:40:38.131 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe708003520 con 0x7fe7200af8e0 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213810748 0 0) 0x7fe7200aed10 con 0x7fe72013ef30 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7200afd50 con 0x7fe72013ef30 2024-03-29T11:40:38.132 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946374192 0 0) 0x7fe708003520 con 0x7fe7200af8e0 2024-03-29T11:40:38.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 >> v1:172.21.15.81:6789/0 conn(0x7fe72013ef30 legacy=0x7fe7200af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.128+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 >> v1:172.21.15.71:6789/0 conn(0x7fe7200b0860 legacy=0x7fe72013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:38.133 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.129+0000 7fe71effd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 1 monclient: found mon.noname-b 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200ab1b0 con 0x7fe7200af8e0 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 10 monclient: _finish_auth 0 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.130+0000 7fe71effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:38.134 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe710001690 con 0x7fe7200af8e0 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: dump: 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7100017f0 con 0x7fe7200af8e0 2024-03-29T11:40:38.135 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 1 -- 172.21.15.71:0/1012491453 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe710001c00 con 0x7fe7200af8e0 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.131+0000 7fe71effd640 10 monclient: dump: 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:38.136 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:38.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.133+0000 7fe726298640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:40:38.137 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.133+0000 7fe726298640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:38.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.133+0000 7fe726298640 4 set_mon_vals no callback set 2024-03-29T11:40:38.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.134+0000 7fe726298640 10 monclient: shutdown 2024-03-29T11:40:38.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.134+0000 7fe726298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:38.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.134+0000 7fe726298640 1 -- 172.21.15.71:0/1012491453 >> v1:172.21.15.71:6790/0 conn(0x7fe7200af8e0 legacy=0x7fe720149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.134+0000 7fe726298640 1 -- 172.21.15.71:0/1012491453 shutdown_connections 2024-03-29T11:40:38.138 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.134+0000 7fe726298640 1 -- 172.21.15.71:0/1012491453 >> 172.21.15.71:0/1012491453 conn(0x7fe72006b660 msgr2=0x7fe72006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:38.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.136+0000 7fe726298640 1 -- 172.21.15.71:0/1012491453 shutdown_connections 2024-03-29T11:40:38.139 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.136+0000 7fe726298640 1 -- 172.21.15.71:0/1012491453 wait complete. 2024-03-29T11:40:38.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.137+0000 7fe726298640 10 monclient: build_initial_monmap 2024-03-29T11:40:38.141 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.137+0000 7fe726298640 10 monclient: monmap: 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.138+0000 7fe726298640 1 librados: starting msgr at 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.138+0000 7fe726298640 1 librados: starting objecter 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.138+0000 7fe726298640 1 Processor -- start 2024-03-29T11:40:38.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.138+0000 7fe726298640 1 -- start start 2024-03-29T11:40:38.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.139+0000 7fe726298640 1 librados: setting wanted keys 2024-03-29T11:40:38.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.139+0000 7fe726298640 1 librados: calling monclient init 2024-03-29T11:40:38.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.139+0000 7fe726298640 10 monclient: init 2024-03-29T11:40:38.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe726298640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:38.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe726298640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:38.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe726298640 10 monclient(hunting): picked mon.noname-b con 0x7fe7200af8e0 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:38.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe726298640 10 monclient(hunting): picked mon.noname-a con 0x7fe7200b0860 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe71ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe7200af8e0 0x7fe7201d6440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:60784/0 (socket says 172.21.15.71:60784) 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe71ffff640 1 -- 172.21.15.71:0/1123859879 learned_addr learned my addr 172.21.15.71:0/1123859879 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe71cff9640 10 client.?.objecter ms_handle_connect 0x7fe7200af8e0 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.140+0000 7fe71cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe71cff9640 10 client.?.objecter ms_handle_connect 0x7fe7200b0860 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe71cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe726298640 10 monclient(hunting): picked mon.noname-c con 0x7fe72013ef30 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe71cff9640 10 client.?.objecter ms_handle_connect 0x7fe72013ef30 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe720135bf0 con 0x7fe7200b0860 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe71cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7200b30a0 con 0x7fe7200af8e0 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7201d6b50 con 0x7fe72013ef30 2024-03-29T11:40:38.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe726298640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe726298640 10 monclient(hunting): authenticate will time out at 1292.296826s 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258380636 0 0) 0x7fe720135bf0 con 0x7fe7200b0860 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.141+0000 7fe71cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe70c004eb0 con 0x7fe7200b0860 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151495158 0 0) 0x7fe7201d6b50 con 0x7fe72013ef30 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720135bf0 con 0x7fe72013ef30 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425620999 0 0) 0x7fe720135bf0 con 0x7fe72013ef30 2024-03-29T11:40:38.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7201d6b50 con 0x7fe72013ef30 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe710001b00 con 0x7fe72013ef30 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.142+0000 7fe71cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 10 monclient(hunting): dump: 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:38.154 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263053577 0 0) 0x7fe7200b30a0 con 0x7fe7200af8e0 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe720135bf0 con 0x7fe7200af8e0 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1258219067 0 0) 0x7fe70c004eb0 con 0x7fe7200b0860 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7200b30a0 con 0x7fe7200b0860 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 99854983 0 0) 0x7fe7201d6b50 con 0x7fe72013ef30 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 >> v1:172.21.15.81:6789/0 conn(0x7fe7200af8e0 legacy=0x7fe7201d6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.143+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 >> v1:172.21.15.71:6789/0 conn(0x7fe7200b0860 legacy=0x7fe7200b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:38.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 1 monclient: found mon.c 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7200b2bc0 con 0x7fe72013ef30 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 10 monclient: _renew_subs 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe720134fb0 con 0x7fe72013ef30 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 10 monclient: _renew_subs 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:38.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72013ac60 con 0x7fe72013ef30 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe726298640 1 librados: init done 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe710004a90 con 0x7fe72013ef30 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71e7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe710001c60 con 0x7fe72013ef30 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.144+0000 7fe71cff9640 10 monclient: dump: 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:38.157 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.145+0000 7fe726298640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.145+0000 7fe726298640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.145+0000 7fe726298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.145+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe720075f10 con 0x7fe72013ef30 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.146+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fe71001ac10 con 0x7fe72013ef30 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71ffff640 10 monclient: get_auth_request con 0x7fe70c06dd90 auth_method 0 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fe710084950 con 0x7fe72013ef30 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71cff9640 10 client.4589.objecter ms_dispatch 0x7fe720000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:38.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71cff9640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:38.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71cff9640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:38.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71cff9640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:40:38.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.147+0000 7fe71cff9640 10 client.4589.objecter ms_handle_connect 0x7fe70c06dd90 2024-03-29T11:40:38.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.154+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe710050060 con 0x7fe72013ef30 2024-03-29T11:40:38.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.154+0000 7fe71cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:38.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.154+0000 7fe71cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:38.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.526+0000 7fe726298640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:38.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.526+0000 7fe726298640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:38.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.526+0000 7fe726298640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:40:38.529 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.526+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7201388c0 con 0x7fe72013ef30 2024-03-29T11:40:38.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.527+0000 7fe71cff9640 1 -- 172.21.15.71:0/1123859879 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fe710052160 con 0x7fe72013ef30 2024-03-29T11:40:38.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.527+0000 7fe71cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:38.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.527+0000 7fe71cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:38.530 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:38.531 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:38.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.530+0000 7fe726298640 10 librados: watch_flush enter 2024-03-29T11:40:38.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 10 librados: watch_flush exit 2024-03-29T11:40:38.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:40:38.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:40:38.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 >> v1:172.21.15.71:6816/1596514235 conn(0x7fe70c06dd90 legacy=0x7fe70c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 10 monclient: shutdown 2024-03-29T11:40:38.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:38.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 >> v1:172.21.15.71:6790/0 conn(0x7fe72013ef30 legacy=0x7fe7201348a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 shutdown_connections 2024-03-29T11:40:38.534 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 >> 172.21.15.71:0/1123859879 conn(0x7fe72006b660 msgr2=0x7fe72013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:38.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 shutdown_connections 2024-03-29T11:40:38.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 -- 172.21.15.71:0/1123859879 wait complete. 2024-03-29T11:40:38.535 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:38.531+0000 7fe726298640 1 librados: shutdown 2024-03-29T11:40:38.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:45.356 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:45.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.500+0000 7fe7ad3ab640 10 monclient: get_monmap_and_config 2024-03-29T11:40:45.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.500+0000 7fe7ad3ab640 10 monclient: build_initial_monmap 2024-03-29T11:40:45.502 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.500+0000 7fe7ad3ab640 10 monclient: monmap: 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:45.501332+0000 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:45.501332+0000 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:45.503 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.501+0000 7fe7ad3ab640 1 Processor -- start 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.501+0000 7fe7ad3ab640 1 -- start start 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.501+0000 7fe7ad3ab640 10 monclient: init 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a0038a10 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a0038df0 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a0030690 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00c54f0 con 0x7fe7a0038a10 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a0036460 con 0x7fe7a0038df0 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a0039870 con 0x7fe7a0030690 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:45.504 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.502+0000 7fe7ad3ab640 10 monclient(hunting): authenticate will time out at 1299.656173s 2024-03-29T11:40:45.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.504+0000 7fe7a77fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe7a0030690 0x7fe7a00d31a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:36742/0 (socket says 172.21.15.71:36742) 2024-03-29T11:40:45.507 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.504+0000 7fe7a77fe640 1 -- 172.21.15.71:0/2758227970 learned_addr learned my addr 172.21.15.71:0/2758227970 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:45.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492762606 0 0) 0x7fe7a0039870 con 0x7fe7a0030690 2024-03-29T11:40:45.512 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe790003520 con 0x7fe7a0030690 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644655617 0 0) 0x7fe7a0036460 con 0x7fe7a0038df0 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a0039870 con 0x7fe7a0038df0 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149641567 0 0) 0x7fe7a00c54f0 con 0x7fe7a0038a10 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.513 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.506+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a0036460 con 0x7fe7a0038a10 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 901646256 0 0) 0x7fe7a0036460 con 0x7fe7a0038a10 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 >> v1:172.21.15.71:6790/0 conn(0x7fe7a0030690 legacy=0x7fe7a00d31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 >> v1:172.21.15.81:6789/0 conn(0x7fe7a0038df0 legacy=0x7fe7a00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a00356c0 con 0x7fe7a0038a10 2024-03-29T11:40:45.514 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe798001630 con 0x7fe7a0038a10 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: dump: 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:45.515 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7ad3ab640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7ad3ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe798001800 con 0x7fe7a0038a10 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 1 -- 172.21.15.71:0/2758227970 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe798001b90 con 0x7fe7a0038a10 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7a5ffb640 10 monclient: dump: 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.516 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7ad3ab640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.507+0000 7fe7ad3ab640 4 set_mon_vals no callback set 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.508+0000 7fe7ad3ab640 10 monclient: shutdown 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.508+0000 7fe7ad3ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.508+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/2758227970 >> v1:172.21.15.71:6789/0 conn(0x7fe7a0038a10 legacy=0x7fe7a0033550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.508+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/2758227970 shutdown_connections 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.508+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/2758227970 >> 172.21.15.71:0/2758227970 conn(0x7fe7a0005c30 msgr2=0x7fe7a0006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:45.517 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.511+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/2758227970 shutdown_connections 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.511+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/2758227970 wait complete. 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.512+0000 7fe7ad3ab640 10 monclient: build_initial_monmap 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.512+0000 7fe7ad3ab640 10 monclient: monmap: 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:45.520 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.512+0000 7fe7ad3ab640 1 librados: starting msgr at 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.512+0000 7fe7ad3ab640 1 librados: starting objecter 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.513+0000 7fe7ad3ab640 1 Processor -- start 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.513+0000 7fe7ad3ab640 1 -- start start 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.513+0000 7fe7ad3ab640 1 librados: setting wanted keys 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.513+0000 7fe7ad3ab640 1 librados: calling monclient init 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.513+0000 7fe7ad3ab640 10 monclient: init 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a0030690 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a0038a10 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:45.521 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a00c96c0 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00c6ac0 con 0x7fe7a0038a10 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00c2fb0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7a00a5540 con 0x7fe7a0030690 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.514+0000 7fe7ad3ab640 10 monclient(hunting): authenticate will time out at 1299.668326s 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe7a6ffd640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe7a0030690 0x7fe7a017ed40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51642/0 (socket says 172.21.15.71:51642) 2024-03-29T11:40:45.522 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe7a6ffd640 1 -- 172.21.15.71:0/3586222941 learned_addr learned my addr 172.21.15.71:0/3586222941 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7a0030690 2024-03-29T11:40:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87730354 0 0) 0x7fe7a00a5540 con 0x7fe7a0030690 2024-03-29T11:40:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.523 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.516+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe794004eb0 con 0x7fe7a0030690 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7a00c96c0 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101335057 0 0) 0x7fe7a00c2fb0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a00a5540 con 0x7fe7a00c96c0 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714061200 0 0) 0x7fe794004eb0 con 0x7fe7a0030690 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7a00c2fb0 con 0x7fe7a0030690 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7980015f0 con 0x7fe7a0030690 2024-03-29T11:40:45.524 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 client.?.objecter ms_handle_connect 0x7fe7a0038a10 2024-03-29T11:40:45.525 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038100735 0 0) 0x7fe7a00a5540 con 0x7fe7a00c96c0 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe794004eb0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7a804fdd0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.521+0000 7fe78f7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:45.526 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:45.527 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:45.527 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910186298 0 0) 0x7fe794004eb0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 >> v1:172.21.15.71:6790/0 conn(0x7fe7a0030690 legacy=0x7fe7a017ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 >> v1:172.21.15.71:6789/0 conn(0x7fe7a0038a10 legacy=0x7fe7a00c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 monclient: found mon.b 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:45.530 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a00a5820 con 0x7fe7a00c96c0 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7a804f400 con 0x7fe7a00c96c0 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe7a8075710 con 0x7fe7a00c96c0 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe78f7fe640 10 monclient: dump: 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.531 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.522+0000 7fe7a57fa640 4 set_mon_vals no callback set 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.523+0000 7fe7ad3ab640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.523+0000 7fe7ad3ab640 10 monclient: _renew_subs 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.523+0000 7fe7ad3ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.523+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a017f450 con 0x7fe7a00c96c0 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.524+0000 7fe7ad3ab640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.524+0000 7fe7ad3ab640 10 monclient: _renew_subs 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.524+0000 7fe7ad3ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:45.532 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.524+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a00ca2b0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.524+0000 7fe7ad3ab640 1 librados: init done 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.525+0000 7fe7ad3ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.525+0000 7fe7ad3ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.525+0000 7fe7ad3ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.525+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a0007ba0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.526+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fe7a8067bf0 con 0x7fe7a00c96c0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.526+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fe7a80ee460 con 0x7fe7a00c96c0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.526+0000 7fe78f7fe640 10 client.4522.objecter ms_dispatch 0x7fe7a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.526+0000 7fe78f7fe640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.526+0000 7fe78f7fe640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.527+0000 7fe78f7fe640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.528+0000 7fe7a6ffd640 10 monclient: get_auth_request con 0x7fe79406dd10 auth_method 0 2024-03-29T11:40:45.533 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.528+0000 7fe78f7fe640 10 client.4522.objecter ms_handle_connect 0x7fe79406dd10 2024-03-29T11:40:45.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.537+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7a80b9d40 con 0x7fe7a00c96c0 2024-03-29T11:40:45.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.537+0000 7fe78f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:45.539 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.537+0000 7fe78f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.866+0000 7fe7ad3ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.866+0000 7fe7ad3ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.866+0000 7fe7ad3ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.866+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7a0005150 con 0x7fe7a00c96c0 2024-03-29T11:40:45.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.867+0000 7fe78f7fe640 1 -- 172.21.15.71:0/3586222941 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fe7a80b9d40 con 0x7fe7a00c96c0 2024-03-29T11:40:45.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.867+0000 7fe78f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:45.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.867+0000 7fe78f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:45.870 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:45.871 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 10 librados: watch_flush enter 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 10 librados: watch_flush exit 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 >> v1:172.21.15.71:6816/1596514235 conn(0x7fe79406dd10 legacy=0x7fe7940701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 10 monclient: shutdown 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 >> v1:172.21.15.81:6789/0 conn(0x7fe7a00c96c0 legacy=0x7fe7a00c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 shutdown_connections 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.872+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 >> 172.21.15.71:0/3586222941 conn(0x7fe7a0005c30 msgr2=0x7fe7a0034a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.874+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 shutdown_connections 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.876+0000 7fe7ad3ab640 1 -- 172.21.15.71:0/3586222941 wait complete. 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:45.876+0000 7fe7ad3ab640 1 librados: shutdown 2024-03-29T11:40:45.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:52.708 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:52.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.840+0000 7f5d4901e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:52.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.840+0000 7f5d4901e640 10 monclient: build_initial_monmap 2024-03-29T11:40:52.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.840+0000 7f5d4901e640 10 monclient: monmap: 2024-03-29T11:40:52.842 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:40:52.841425+0000 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:40:52.841425+0000 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:40:52.843 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.841+0000 7f5d4901e640 1 Processor -- start 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.841+0000 7f5d4901e640 1 -- start start 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.841+0000 7f5d4901e640 10 monclient: init 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient(hunting): picked mon.noname-b con 0x7f5d441457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d42d9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5d441457f0 0x7f5d4413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50338/0 (socket says 172.21.15.71:50338) 2024-03-29T11:40:52.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient(hunting): picked mon.noname-c con 0x7f5d440a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d42d9d640 1 -- 172.21.15.71:0/2485127768 learned_addr learned my addr 172.21.15.71:0/2485127768 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d440aaef0 con 0x7f5d4413f650 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44145bd0 con 0x7f5d440a9d70 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44145eb0 con 0x7f5d441457f0 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.842+0000 7f5d4901e640 10 monclient(hunting): authenticate will time out at 1306.997008s 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4921540 0 0) 0x7f5d44145eb0 con 0x7f5d441457f0 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:52.846 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:52.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:40:52.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c003520 con 0x7f5d441457f0 2024-03-29T11:40:52.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 724866472 0 0) 0x7f5d2c003520 con 0x7f5d441457f0 2024-03-29T11:40:52.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 >> v1:172.21.15.81:6789/0 conn(0x7f5d440a9d70 legacy=0x7f5d4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:52.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.843+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 >> v1:172.21.15.71:6789/0 conn(0x7f5d4413f650 legacy=0x7f5d440b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:52.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:40:52.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 1 monclient: found mon.noname-b 2024-03-29T11:40:52.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:52.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:40:52.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d44146340 con 0x7f5d441457f0 2024-03-29T11:40:52.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: _finish_auth 0 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5d34001690 con 0x7f5d441457f0 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d41d9b640 10 monclient: dump: 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:52.850 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d4901e640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.845+0000 7f5d4901e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d340018f0 con 0x7f5d441457f0 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d41d9b640 1 -- 172.21.15.71:0/2485127768 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5d34001d40 con 0x7f5d441457f0 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d41d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d41d9b640 10 monclient: dump: 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.851 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d4901e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:52.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.846+0000 7f5d4901e640 4 set_mon_vals no callback set 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 10 monclient: shutdown 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 >> v1:172.21.15.71:6790/0 conn(0x7f5d441457f0 legacy=0x7f5d4413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 shutdown_connections 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 >> 172.21.15.71:0/2485127768 conn(0x7f5d44066a00 msgr2=0x7f5d44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 shutdown_connections 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.848+0000 7f5d4901e640 1 -- 172.21.15.71:0/2485127768 wait complete. 2024-03-29T11:40:52.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.849+0000 7f5d4901e640 10 monclient: build_initial_monmap 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.849+0000 7f5d4901e640 10 monclient: monmap: 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.849+0000 7f5d4901e640 1 librados: starting msgr at 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.849+0000 7f5d4901e640 1 librados: starting objecter 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.851+0000 7f5d4901e640 1 Processor -- start 2024-03-29T11:40:52.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.851+0000 7f5d4901e640 1 -- start start 2024-03-29T11:40:52.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.851+0000 7f5d4901e640 1 librados: setting wanted keys 2024-03-29T11:40:52.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.851+0000 7f5d4901e640 1 librados: calling monclient init 2024-03-29T11:40:52.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.851+0000 7f5d4901e640 10 monclient: init 2024-03-29T11:40:52.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.852+0000 7f5d4901e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:52.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.852+0000 7f5d4901e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:52.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.852+0000 7f5d4901e640 10 monclient(hunting): picked mon.noname-b con 0x7f5d440a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:40:52.862 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 10 monclient(hunting): picked mon.noname-c con 0x7f5d441457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44142ce0 con 0x7f5d4413f650 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d44140870 con 0x7f5d440a9d70 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d441416e0 con 0x7f5d441457f0 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.853+0000 7f5d4901e640 10 monclient(hunting): authenticate will time out at 1307.007921s 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d4359e640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5d441457f0 0x7f5d441425d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50954/0 (socket says 172.21.15.71:50954) 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d4359e640 1 -- 172.21.15.71:0/3224197491 learned_addr learned my addr 172.21.15.71:0/3224197491 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 client.?.objecter ms_handle_connect 0x7f5d441457f0 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 client.?.objecter ms_handle_connect 0x7f5d440a9d70 2024-03-29T11:40:52.863 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096159038 0 0) 0x7f5d44140870 con 0x7f5d440a9d70 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.854+0000 7f5d2b7fe640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d30004eb0 con 0x7f5d440a9d70 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613582267 0 0) 0x7f5d441416e0 con 0x7f5d441457f0 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d300051d0 con 0x7f5d441457f0 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2406006741 0 0) 0x7f5d30004eb0 con 0x7f5d440a9d70 2024-03-29T11:40:52.864 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d441416e0 con 0x7f5d440a9d70 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 626725445 0 0) 0x7f5d300051d0 con 0x7f5d441457f0 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d30005a70 con 0x7f5d441457f0 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5d3c004340 con 0x7f5d441457f0 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:52.865 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5d34001650 con 0x7f5d440a9d70 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.855+0000 7f5d2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.866 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1587807288 0 0) 0x7f5d441416e0 con 0x7f5d440a9d70 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 >> v1:172.21.15.71:6790/0 conn(0x7f5d441457f0 legacy=0x7f5d441425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 >> v1:172.21.15.71:6789/0 conn(0x7f5d4413f650 legacy=0x7f5d44141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 monclient: found mon.b 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4413e7b0 con 0x7f5d440a9d70 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:52.867 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 10 monclient: _renew_subs 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d44130b30 con 0x7f5d440a9d70 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 10 monclient: _renew_subs 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4413be20 con 0x7f5d440a9d70 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4901e640 1 librados: init done 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d34004a90 con 0x7f5d440a9d70 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5d34001650 con 0x7f5d440a9d70 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:52.868 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d2b7fe640 10 monclient: dump: 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.856+0000 7f5d4159a640 4 set_mon_vals no callback set 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.858+0000 7f5d4901e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.858+0000 7f5d4901e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.858+0000 7f5d4901e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.858+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d24005080 con 0x7f5d440a9d70 2024-03-29T11:40:52.869 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.859+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f5d3401aa70 con 0x7f5d440a9d70 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f5d340847b0 con 0x7f5d440a9d70 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d2b7fe640 10 client.4525.objecter ms_dispatch 0x7f5d44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d2b7fe640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d2b7fe640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d2b7fe640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d4259c640 10 monclient: get_auth_request con 0x7f5d3006e480 auth_method 0 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.860+0000 7f5d2b7fe640 10 client.4525.objecter ms_handle_connect 0x7f5d3006e480 2024-03-29T11:40:52.870 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.867+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d34050090 con 0x7f5d440a9d70 2024-03-29T11:40:52.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.867+0000 7f5d2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:52.871 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:52.867+0000 7f5d2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:53.177 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.175+0000 7f5d4901e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.175+0000 7f5d4901e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.175+0000 7f5d4901e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.175+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d24005330 con 0x7f5d440a9d70 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.176+0000 7f5d2b7fe640 1 -- 172.21.15.71:0/3224197491 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f5d34051fc0 con 0x7f5d440a9d70 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.176+0000 7f5d2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.176+0000 7f5d2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.179+0000 7f5d4901e640 10 librados: watch_flush enter 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 10 librados: watch_flush exit 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 >> v1:172.21.15.71:6816/1596514235 conn(0x7f5d3006e480 legacy=0x7f5d30070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 10 monclient: shutdown 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 >> v1:172.21.15.81:6789/0 conn(0x7f5d440a9d70 legacy=0x7f5d44140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 shutdown_connections 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 >> 172.21.15.71:0/3224197491 conn(0x7f5d44066a00 msgr2=0x7f5d440b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 shutdown_connections 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 -- 172.21.15.71:0/3224197491 wait complete. 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:40:53.180+0000 7f5d4901e640 1 librados: shutdown 2024-03-29T11:40:53.196 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:00.003 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.137+0000 7f403f0f1640 10 monclient: get_monmap_and_config 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.137+0000 7f403f0f1640 10 monclient: build_initial_monmap 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.138+0000 7f403f0f1640 10 monclient: monmap: 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:00.138899+0000 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:00.138899+0000 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:00.140 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.138+0000 7f403f0f1640 1 Processor -- start 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.138+0000 7f403f0f1640 1 -- start start 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.138+0000 7f403f0f1640 10 monclient: init 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.139+0000 7f403f0f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.139+0000 7f403f0f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.139+0000 7f403f0f1640 10 monclient(hunting): picked mon.noname-b con 0x7f40381422b0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 10 monclient(hunting): picked mon.noname-c con 0x7f4038142690 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 10 monclient(hunting): picked mon.noname-a con 0x7f40380b17d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4038142c80 con 0x7f40380b17d0 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380aae40 con 0x7f4038142690 2024-03-29T11:41:00.142 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403813fbd0 con 0x7f40381422b0 2024-03-29T11:41:00.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:00.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403f0f1640 10 monclient(hunting): authenticate will time out at 1314.294242s 2024-03-29T11:41:00.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403ce8d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f40381422b0 0x7f403813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48046/0 (socket says 172.21.15.71:48046) 2024-03-29T11:41:00.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f403ce8d640 1 -- 172.21.15.71:0/3884614617 learned_addr learned my addr 172.21.15.71:0/3884614617 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:00.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512599591 0 0) 0x7f403813fbd0 con 0x7f40381422b0 2024-03-29T11:41:00.143 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f40377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f40377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f40377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f40377fe640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.140+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4028003520 con 0x7f40381422b0 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.141+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131499912 0 0) 0x7f4028003520 con 0x7f40381422b0 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.141+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 >> v1:172.21.15.81:6789/0 conn(0x7f4038142690 legacy=0x7f40380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.144 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.141+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 >> v1:172.21.15.71:6789/0 conn(0x7f40380b17d0 legacy=0x7f40380b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.142+0000 7f40377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.142+0000 7f40377fe640 1 monclient: found mon.noname-b 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.142+0000 7f40377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.142+0000 7f40377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.142+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4038151940 con 0x7f40381422b0 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:00.145 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f402c001690 con 0x7f40381422b0 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f403f0f1640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f403f0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: dump: 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:00.146 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:00.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f402c0018b0 con 0x7f40381422b0 2024-03-29T11:41:00.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f403f0f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:00.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:00.147 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 1 -- 172.21.15.71:0/3884614617 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f402c001c40 con 0x7f40381422b0 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f403f0f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f403f0f1640 4 set_mon_vals no callback set 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.143+0000 7f40377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.144+0000 7f40377fe640 10 monclient: dump: 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:00.148 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.144+0000 7f403f0f1640 10 monclient: shutdown 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.144+0000 7f403f0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.144+0000 7f403f0f1640 1 -- 172.21.15.71:0/3884614617 >> v1:172.21.15.71:6790/0 conn(0x7f40381422b0 legacy=0x7f403813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.145+0000 7f403f0f1640 1 -- 172.21.15.71:0/3884614617 shutdown_connections 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.145+0000 7f403f0f1640 1 -- 172.21.15.71:0/3884614617 >> 172.21.15.71:0/3884614617 conn(0x7f4038066a00 msgr2=0x7f4038066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.145+0000 7f403f0f1640 1 -- 172.21.15.71:0/3884614617 shutdown_connections 2024-03-29T11:41:00.149 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.145+0000 7f403f0f1640 1 -- 172.21.15.71:0/3884614617 wait complete. 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.146+0000 7f403f0f1640 10 monclient: build_initial_monmap 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.146+0000 7f403f0f1640 10 monclient: monmap: 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:00.150 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 1 librados: starting msgr at 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 1 librados: starting objecter 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 1 Processor -- start 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 1 -- start start 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 1 librados: setting wanted keys 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 1 librados: calling monclient init 2024-03-29T11:41:00.151 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.148+0000 7f403f0f1640 10 monclient: init 2024-03-29T11:41:00.152 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient(hunting): picked mon.noname-c con 0x7f40380b17d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient(hunting): picked mon.noname-b con 0x7f40381422b0 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient(hunting): picked mon.noname-a con 0x7f4038142690 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380ad060 con 0x7f4038142690 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4038132510 con 0x7f40381422b0 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4038131fe0 con 0x7f40380b17d0 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403f0f1640 10 monclient(hunting): authenticate will time out at 1314.303773s 2024-03-29T11:41:00.153 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403ce8d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f40380b17d0 0x7f40380ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49408/0 (socket says 172.21.15.71:49408) 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f403ce8d640 1 -- 172.21.15.71:0/106476969 learned_addr learned my addr 172.21.15.71:0/106476969 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f40357fa640 10 client.?.objecter ms_handle_connect 0x7f40380b17d0 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.149+0000 7f40357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 client.?.objecter ms_handle_connect 0x7f40381422b0 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622100318 0 0) 0x7f4038131fe0 con 0x7f40380b17d0 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c004eb0 con 0x7f40380b17d0 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102031850 0 0) 0x7f401c004eb0 con 0x7f40380b17d0 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4038131fe0 con 0x7f40380b17d0 2024-03-29T11:41:00.154 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f402c001550 con 0x7f40380b17d0 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): dump: 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:00.155 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3802166779 0 0) 0x7f4038131fe0 con 0x7f40380b17d0 2024-03-29T11:41:00.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 >> v1:172.21.15.81:6789/0 conn(0x7f40381422b0 legacy=0x7f40381dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.156 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.150+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 >> v1:172.21.15.71:6789/0 conn(0x7f4038142690 legacy=0x7f40381db240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:00.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:00.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 1 monclient: found mon.c 2024-03-29T11:41:00.157 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380abec0 con 0x7f40380b17d0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f40357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 10 monclient: _renew_subs 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40381e2de0 con 0x7f40380b17d0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 10 monclient: _renew_subs 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40381e3150 con 0x7f40380b17d0 2024-03-29T11:41:00.158 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.153+0000 7f403f0f1640 1 librados: init done 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f402c004bf0 con 0x7f40380b17d0 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f40357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f402c004f00 con 0x7f40380b17d0 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f40357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f40357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f40357fa640 10 monclient: dump: 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:00.159 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.154+0000 7f4036ffd640 4 set_mon_vals no callback set 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.155+0000 7f403f0f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.155+0000 7f403f0f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.155+0000 7f403f0f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.155+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4018005080 con 0x7f40380b17d0 2024-03-29T11:41:00.160 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.155+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f402c01aed0 con 0x7f40380b17d0 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f4037fff640 10 monclient: get_auth_request con 0x7f401c06dc60 auth_method 0 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f402c084bd0 con 0x7f40380b17d0 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f40357fa640 10 client.4607.objecter ms_dispatch 0x7f4038000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f40357fa640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f40357fa640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f40357fa640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:41:00.161 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.156+0000 7f40357fa640 10 client.4607.objecter ms_handle_connect 0x7f401c06dc60 2024-03-29T11:41:00.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.163+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f402c050510 con 0x7f40380b17d0 2024-03-29T11:41:00.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.163+0000 7f40357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:00.165 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.163+0000 7f40357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.475+0000 7f403f0f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.475+0000 7f403f0f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.475+0000 7f403f0f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.475+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4018005330 con 0x7f40380b17d0 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.476+0000 7f40357fa640 1 -- 172.21.15.71:0/106476969 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f402c050510 con 0x7f40380b17d0 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.476+0000 7f40357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:00.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.476+0000 7f40357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:00.479 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:00.479 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 10 librados: watch_flush enter 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 10 librados: watch_flush exit 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 >> v1:172.21.15.71:6816/1596514235 conn(0x7f401c06dc60 legacy=0x7f401c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 10 monclient: shutdown 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.480+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 >> v1:172.21.15.71:6790/0 conn(0x7f40380b17d0 legacy=0x7f40380ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.481+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 shutdown_connections 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.481+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 >> 172.21.15.71:0/106476969 conn(0x7f4038066a00 msgr2=0x7f403813d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:00.483 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.481+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 shutdown_connections 2024-03-29T11:41:00.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.481+0000 7f403f0f1640 1 -- 172.21.15.71:0/106476969 wait complete. 2024-03-29T11:41:00.484 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:00.481+0000 7f403f0f1640 1 librados: shutdown 2024-03-29T11:41:00.499 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:07.305 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.451+0000 7f51ee815640 10 monclient: get_monmap_and_config 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.451+0000 7f51ee815640 10 monclient: build_initial_monmap 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.451+0000 7f51ee815640 10 monclient: monmap: 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:07.452465+0000 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:07.452465+0000 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:07.455 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:07.456 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:07.456 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:07.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.453+0000 7f51ee815640 1 Processor -- start 2024-03-29T11:41:07.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.453+0000 7f51ee815640 1 -- start start 2024-03-29T11:41:07.456 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.453+0000 7f51ee815640 10 monclient: init 2024-03-29T11:41:07.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.454+0000 7f51ee815640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:07.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.454+0000 7f51ee815640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:07.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.454+0000 7f51ee815640 10 monclient(hunting): picked mon.noname-b con 0x7f51e80a9ac0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:07.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.454+0000 7f51ee815640 10 monclient(hunting): picked mon.noname-c con 0x7f51e813ee90 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:07.458 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.454+0000 7f51e7fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f51e80a9ac0 0x7f51e80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:33602/0 (socket says 172.21.15.71:33602) 2024-03-29T11:41:07.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51ee815640 10 monclient(hunting): picked mon.noname-a con 0x7f51e80af900 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:07.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51ee815640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80afd10 con 0x7f51e80af900 2024-03-29T11:41:07.459 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51ee815640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80af6f0 con 0x7f51e813ee90 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51ee815640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e80aac80 con 0x7f51e80a9ac0 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51ee815640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51ee815640 10 monclient(hunting): authenticate will time out at 1321.610484s 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e7fff640 1 -- 172.21.15.71:0/983675742 learned_addr learned my addr 172.21.15.71:0/983675742 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571555248 0 0) 0x7f51e80aac80 con 0x7f51e80a9ac0 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e6ffd640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.456+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d0003520 con 0x7f51e80a9ac0 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.457+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 51227348 0 0) 0x7f51d0003520 con 0x7f51e80a9ac0 2024-03-29T11:41:07.460 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.457+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 >> v1:172.21.15.81:6789/0 conn(0x7f51e813ee90 legacy=0x7f51e813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:07.461 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 >> v1:172.21.15.71:6789/0 conn(0x7f51e80af900 legacy=0x7f51e8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e80aaf60 con 0x7f51e80a9ac0 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:07.462 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f51dc001690 con 0x7f51e80a9ac0 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: dump: 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:07.463 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51dc001960 con 0x7f51e80a9ac0 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 1 -- 172.21.15.71:0/983675742 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f51dc001cf0 con 0x7f51e80a9ac0 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.459+0000 7f51e6ffd640 10 monclient: dump: 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:07.464 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:07.465 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:07.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.460+0000 7f51ee815640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:41:07.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.461+0000 7f51ee815640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:07.465 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.461+0000 7f51ee815640 4 set_mon_vals no callback set 2024-03-29T11:41:07.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.461+0000 7f51ee815640 10 monclient: shutdown 2024-03-29T11:41:07.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.461+0000 7f51ee815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:07.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.461+0000 7f51ee815640 1 -- 172.21.15.71:0/983675742 >> v1:172.21.15.71:6790/0 conn(0x7f51e80a9ac0 legacy=0x7f51e80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.462+0000 7f51ee815640 1 -- 172.21.15.71:0/983675742 shutdown_connections 2024-03-29T11:41:07.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.462+0000 7f51ee815640 1 -- 172.21.15.71:0/983675742 >> 172.21.15.71:0/983675742 conn(0x7f51e80b35e0 msgr2=0x7f51e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:07.466 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.463+0000 7f51ee815640 1 -- 172.21.15.71:0/983675742 shutdown_connections 2024-03-29T11:41:07.467 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.466+0000 7f51ee815640 1 -- 172.21.15.71:0/983675742 wait complete. 2024-03-29T11:41:07.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.468+0000 7f51ee815640 10 monclient: build_initial_monmap 2024-03-29T11:41:07.470 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.468+0000 7f51ee815640 10 monclient: monmap: 2024-03-29T11:41:07.470 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:07.470 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:07.470 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:07.471 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.468+0000 7f51ee815640 1 librados: starting msgr at 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.468+0000 7f51ee815640 1 librados: starting objecter 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.469+0000 7f51ee815640 1 Processor -- start 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.469+0000 7f51ee815640 1 -- start start 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.469+0000 7f51ee815640 1 librados: setting wanted keys 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.469+0000 7f51ee815640 1 librados: calling monclient init 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.469+0000 7f51ee815640 10 monclient: init 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient(hunting): picked mon.noname-b con 0x7f51e80a9ac0 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:07.475 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient(hunting): picked mon.noname-c con 0x7f51e80af900 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient(hunting): picked mon.noname-a con 0x7f51e813ee90 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e8071880 con 0x7f51e813ee90 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e81db2c0 con 0x7f51e80a9ac0 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e81db450 con 0x7f51e80af900 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51ee815640 10 monclient(hunting): authenticate will time out at 1321.624549s 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e77fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f51e80af900 0x7f51e81dabb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34748/0 (socket says 172.21.15.71:34748) 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e77fe640 1 -- 172.21.15.71:0/1815343212 learned_addr learned my addr 172.21.15.71:0/1815343212 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 10 client.?.objecter ms_handle_connect 0x7f51e80af900 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130850604 0 0) 0x7f51e81db450 con 0x7f51e80af900 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:07.476 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.470+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d4004eb0 con 0x7f51e80af900 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2052308462 0 0) 0x7f51d4004eb0 con 0x7f51e80af900 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51d4005300 con 0x7f51e80af900 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f51d8001790 con 0x7f51e80af900 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 10 monclient(hunting): dump: 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.477 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1553178291 0 0) 0x7f51d4005300 con 0x7f51e80af900 2024-03-29T11:41:07.478 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.471+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 >> v1:172.21.15.81:6789/0 conn(0x7f51e80a9ac0 legacy=0x7f51e8071040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:07.479 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.477+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 >> v1:172.21.15.71:6789/0 conn(0x7f51e813ee90 legacy=0x7f51e81e72a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 1 monclient: found mon.c 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e8070a80 con 0x7f51e80af900 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 10 monclient: _renew_subs 2024-03-29T11:41:07.485 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e8070cd0 con 0x7f51e80af900 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 10 monclient: _renew_subs 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e8141f10 con 0x7f51e80af900 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.482+0000 7f51ee815640 1 librados: init done 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51d8001d60 con 0x7f51e80af900 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:07.486 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e5ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f51d8001790 con 0x7f51e80af900 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.483+0000 7f51e4ff9640 10 monclient: dump: 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:07.487 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.485+0000 7f51ee815640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.485+0000 7f51ee815640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.485+0000 7f51ee815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:07.488 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.485+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c8005080 con 0x7f51e80af900 2024-03-29T11:41:07.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.485+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f51d801a810 con 0x7f51e80af900 2024-03-29T11:41:07.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.486+0000 7f51e7fff640 10 monclient: get_auth_request con 0x7f51d406e000 auth_method 0 2024-03-29T11:41:07.489 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.486+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f51d80545f0 con 0x7f51e80af900 2024-03-29T11:41:07.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.486+0000 7f51e4ff9640 10 client.4613.objecter ms_dispatch 0x7f51e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:07.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.486+0000 7f51e4ff9640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:07.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.486+0000 7f51e4ff9640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:07.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.486+0000 7f51e4ff9640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:41:07.490 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.487+0000 7f51e4ff9640 10 client.4613.objecter ms_handle_connect 0x7f51d406e000 2024-03-29T11:41:07.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.495+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51d8050890 con 0x7f51e80af900 2024-03-29T11:41:07.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.495+0000 7f51e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:07.497 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.495+0000 7f51e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:07.826 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.824+0000 7f51ee815640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.824+0000 7f51ee815640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.824+0000 7f51ee815640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:07.827 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.824+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51c8005330 con 0x7f51e80af900 2024-03-29T11:41:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.825+0000 7f51e4ff9640 1 -- 172.21.15.71:0/1815343212 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f51d8050890 con 0x7f51e80af900 2024-03-29T11:41:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.825+0000 7f51e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:07.828 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.825+0000 7f51e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:07.829 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:07.829 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:07.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.828+0000 7f51ee815640 10 librados: watch_flush enter 2024-03-29T11:41:07.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.828+0000 7f51ee815640 10 librados: watch_flush exit 2024-03-29T11:41:07.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.828+0000 7f51ee815640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:41:07.833 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.828+0000 7f51ee815640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 >> v1:172.21.15.71:6816/1596514235 conn(0x7f51d406e000 legacy=0x7f51d40704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 10 monclient: shutdown 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 >> v1:172.21.15.71:6790/0 conn(0x7f51e80af900 legacy=0x7f51e81dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 shutdown_connections 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 >> 172.21.15.71:0/1815343212 conn(0x7f51e80b35e0 msgr2=0x7f51e813b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 shutdown_connections 2024-03-29T11:41:07.834 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 -- 172.21.15.71:0/1815343212 wait complete. 2024-03-29T11:41:07.835 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:07.829+0000 7f51ee815640 1 librados: shutdown 2024-03-29T11:41:07.850 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:14.657 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.826+0000 7ff81ab9f640 10 monclient: get_monmap_and_config 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.826+0000 7ff81ab9f640 10 monclient: build_initial_monmap 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.826+0000 7ff81ab9f640 10 monclient: monmap: 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:14.827355+0000 2024-03-29T11:41:14.830 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:14.827355+0000 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:14.831 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.829+0000 7ff81ab9f640 1 Processor -- start 2024-03-29T11:41:14.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.830+0000 7ff81ab9f640 1 -- start start 2024-03-29T11:41:14.832 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.830+0000 7ff81ab9f640 10 monclient: init 2024-03-29T11:41:14.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:14.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:14.836 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff80c006410 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient(hunting): picked mon.noname-b con 0x7ff80c02a640 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff80c02b370 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c02a120 con 0x7ff80c006410 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0c75b0 con 0x7ff80c02b370 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c033110 con 0x7ff80c02a640 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.831+0000 7ff81ab9f640 10 monclient(hunting): authenticate will time out at 1328.985683s 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.834+0000 7ff81a39e640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff80c02b370 0x7ff80c0320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:51433/0 (socket says 172.21.15.71:51433) 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.834+0000 7ff81a39e640 1 -- 172.21.15.71:0/203240063 learned_addr learned my addr 172.21.15.71:0/203240063 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:14.837 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.834+0000 7ff81939c640 1 --1- 172.21.15.71:0/203240063 >> v1:172.21.15.71:6790/0 conn(0x7ff80c02a640 0x7ff80c0c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:48698/0 (socket says 172.21.15.71:48698) 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050850421 0 0) 0x7ff80c0c75b0 con 0x7ff80c02b370 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff800003520 con 0x7ff80c02b370 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1414179286 0 0) 0x7ff800003520 con 0x7ff80c02b370 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 >> v1:172.21.15.71:6790/0 conn(0x7ff80c02a640 legacy=0x7ff80c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:14.839 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.835+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 >> v1:172.21.15.71:6789/0 conn(0x7ff80c006410 legacy=0x7ff80c0345e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 1 monclient: found mon.noname-c 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0332a0 con 0x7ff80c02b370 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff81ab9f640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:41:14.840 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff810001690 con 0x7ff80c02b370 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: dump: 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:14.841 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff81ab9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8100017f0 con 0x7ff80c02b370 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff818b9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff81ab9f640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.836+0000 7ff81ab9f640 4 set_mon_vals no callback set 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff818b9b640 1 -- 172.21.15.71:0/203240063 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff810001c40 con 0x7ff80c02b370 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff818b9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff818b9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:14.842 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff818b9b640 10 monclient: dump: 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:14.843 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff81ab9f640 10 monclient: shutdown 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff81ab9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff81ab9f640 1 -- 172.21.15.71:0/203240063 >> v1:172.21.15.81:6789/0 conn(0x7ff80c02b370 legacy=0x7ff80c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff81ab9f640 1 -- 172.21.15.71:0/203240063 shutdown_connections 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.837+0000 7ff81ab9f640 1 -- 172.21.15.71:0/203240063 >> 172.21.15.71:0/203240063 conn(0x7ff80c0058f0 msgr2=0x7ff80c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.839+0000 7ff81ab9f640 1 -- 172.21.15.71:0/203240063 shutdown_connections 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.839+0000 7ff81ab9f640 1 -- 172.21.15.71:0/203240063 wait complete. 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.840+0000 7ff81ab9f640 10 monclient: build_initial_monmap 2024-03-29T11:41:14.845 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.840+0000 7ff81ab9f640 10 monclient: monmap: 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:14.846 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.840+0000 7ff81ab9f640 1 librados: starting msgr at 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.840+0000 7ff81ab9f640 1 librados: starting objecter 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.840+0000 7ff81ab9f640 1 Processor -- start 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 1 -- start start 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 1 librados: setting wanted keys 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 1 librados: calling monclient init 2024-03-29T11:41:14.847 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 10 monclient: init 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff80c006410 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 10 monclient(hunting): picked mon.noname-b con 0x7ff80c02a640 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff80c02b370 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.841+0000 7ff81ab9f640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0c2ed0 con 0x7ff80c02b370 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.842+0000 7ff81ab9f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0c49d0 con 0x7ff80c02a640 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.842+0000 7ff81ab9f640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80c0c4cb0 con 0x7ff80c006410 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.842+0000 7ff81ab9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.842+0000 7ff81ab9f640 10 monclient(hunting): authenticate will time out at 1328.996065s 2024-03-29T11:41:14.848 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.842+0000 7ff819b9d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7ff80c006410 0x7ff80c0c27c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:53988/0 (socket says 172.21.15.71:53988) 2024-03-29T11:41:14.849 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.842+0000 7ff819b9d640 1 -- 172.21.15.71:0/1519515070 learned_addr learned my addr 172.21.15.71:0/1519515070 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 10 client.?.objecter ms_handle_connect 0x7ff80c006410 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204086784 0 0) 0x7ff80c0c4cb0 con 0x7ff80c006410 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.843+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f8004eb0 con 0x7ff80c006410 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.844+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415040670 0 0) 0x7ff7f8004eb0 con 0x7ff80c006410 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.844+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff80c0c4cb0 con 0x7ff80c006410 2024-03-29T11:41:14.851 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.845+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff81404fdd0 con 0x7ff80c006410 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.845+0000 7ff80a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.845+0000 7ff80a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.845+0000 7ff80a7fc640 10 monclient(hunting): dump: 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.845+0000 7ff80a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:14.852 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.847+0000 7ff80a7fc640 10 client.?.objecter ms_handle_connect 0x7ff80c02a640 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3718894856 0 0) 0x7ff80c0c4cb0 con 0x7ff80c006410 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 >> v1:172.21.15.81:6789/0 conn(0x7ff80c02a640 legacy=0x7ff80c0c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 >> v1:172.21.15.71:6789/0 conn(0x7ff80c02b370 legacy=0x7ff80c189720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 1 monclient: found mon.c 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80c0bd7a0 con 0x7ff80c006410 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff80a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:41:14.853 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 10 monclient: _renew_subs 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80c0c3a60 con 0x7ff80c006410 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 10 monclient: _renew_subs 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff80c030470 con 0x7ff80c006410 2024-03-29T11:41:14.854 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.848+0000 7ff81ab9f640 1 librados: init done 2024-03-29T11:41:14.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.851+0000 7ff81ab9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:14.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.851+0000 7ff81ab9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:14.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.851+0000 7ff81ab9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:14.855 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.851+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80c0c3110 con 0x7ff80c006410 2024-03-29T11:41:14.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff81404f400 con 0x7ff80c006410 2024-03-29T11:41:14.856 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7ff814075710 con 0x7ff80c006410 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 10 monclient: dump: 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:14.857 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7ff814067bf0 con 0x7ff80c006410 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7ff8140ee5e0 con 0x7ff80c006410 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 10 client.4619.objecter ms_dispatch 0x7ff814000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.852+0000 7ff80a7fc640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.853+0000 7ff80a7fc640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.853+0000 7ff80b7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.854+0000 7ff81939c640 10 monclient: get_auth_request con 0x7ff7f806db60 auth_method 0 2024-03-29T11:41:14.858 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.854+0000 7ff80a7fc640 10 client.4619.objecter ms_handle_connect 0x7ff7f806db60 2024-03-29T11:41:14.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.863+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8140b9f40 con 0x7ff80c006410 2024-03-29T11:41:14.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.863+0000 7ff80a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:14.865 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:14.863+0000 7ff80a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.215+0000 7ff81ab9f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.215+0000 7ff81ab9f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.215+0000 7ff81ab9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:15.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.215+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff80c004cd0 con 0x7ff80c006410 2024-03-29T11:41:15.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.216+0000 7ff80a7fc640 1 -- 172.21.15.71:0/1519515070 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7ff8140bbe70 con 0x7ff80c006410 2024-03-29T11:41:15.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.216+0000 7ff80a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:15.219 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.216+0000 7ff80a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:15.219 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:15.219 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:15.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 10 librados: watch_flush enter 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 10 librados: watch_flush exit 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 10 client.4619.objecter successfully canceled tick 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 >> v1:172.21.15.71:6816/1596514235 conn(0x7ff7f806db60 legacy=0x7ff7f8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 10 monclient: shutdown 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 >> v1:172.21.15.71:6790/0 conn(0x7ff80c006410 legacy=0x7ff80c0c27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 shutdown_connections 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.220+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 >> 172.21.15.71:0/1519515070 conn(0x7ff80c0058f0 msgr2=0x7ff80c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.221+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 shutdown_connections 2024-03-29T11:41:15.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.221+0000 7ff81ab9f640 1 -- 172.21.15.71:0/1519515070 wait complete. 2024-03-29T11:41:15.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:15.221+0000 7ff81ab9f640 1 librados: shutdown 2024-03-29T11:41:15.239 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:22.046 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:22.203 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.196+0000 7f088aa6a640 10 monclient: get_monmap_and_config 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.196+0000 7f088aa6a640 10 monclient: build_initial_monmap 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.196+0000 7f088aa6a640 10 monclient: monmap: 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:22.197404+0000 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:22.197404+0000 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:22.204 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.197+0000 7f088aa6a640 1 Processor -- start 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.197+0000 7f088aa6a640 1 -- start start 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.197+0000 7f088aa6a640 10 monclient: init 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient(hunting): picked mon.noname-c con 0x7f08840ad6c0 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient(hunting): picked mon.noname-b con 0x7f08840af6c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient(hunting): picked mon.noname-a con 0x7f088414fdc0 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08840a94e0 con 0x7f088414fdc0 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08840af3a0 con 0x7f08840ad6c0 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08840afb60 con 0x7f08840af6c0 2024-03-29T11:41:22.205 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:22.206 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.198+0000 7f088aa6a640 10 monclient(hunting): authenticate will time out at 1336.352948s 2024-03-29T11:41:22.212 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.205+0000 7f0888806640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f08840ad6c0 0x7f088414f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:42217/0 (socket says 172.21.15.71:42217) 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.205+0000 7f0888806640 1 -- 172.21.15.71:0/2456808093 learned_addr learned my addr 172.21.15.71:0/2456808093 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.206+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205150745 0 0) 0x7f08840af3a0 con 0x7f08840ad6c0 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.206+0000 7f08837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.206+0000 7f08837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.206+0000 7f08837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.206+0000 7f08837fe640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.206+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f086c003500 con 0x7f08840ad6c0 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 855584730 0 0) 0x7f086c003500 con 0x7f08840ad6c0 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 >> v1:172.21.15.71:6790/0 conn(0x7f08840af6c0 legacy=0x7f0884152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 >> v1:172.21.15.71:6789/0 conn(0x7f088414fdc0 legacy=0x7f088415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 monclient: found mon.noname-c 2024-03-29T11:41:22.213 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840aab80 con 0x7f08840ad6c0 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f088aa6a640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f088aa6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f087c001670 con 0x7f08840ad6c0 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: dump: 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:22.214 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f088aa6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f087c001890 con 0x7f08840ad6c0 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 1 -- 172.21.15.71:0/2456808093 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f087c001c20 con 0x7f08840ad6c0 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f08837fe640 10 monclient: dump: 2024-03-29T11:41:22.215 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:22.216 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f088aa6a640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.207+0000 7f088aa6a640 4 set_mon_vals no callback set 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.208+0000 7f088aa6a640 10 monclient: shutdown 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.208+0000 7f088aa6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.208+0000 7f088aa6a640 1 -- 172.21.15.71:0/2456808093 >> v1:172.21.15.81:6789/0 conn(0x7f08840ad6c0 legacy=0x7f088414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.208+0000 7f088aa6a640 1 -- 172.21.15.71:0/2456808093 shutdown_connections 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.208+0000 7f088aa6a640 1 -- 172.21.15.71:0/2456808093 >> 172.21.15.71:0/2456808093 conn(0x7f08840b35e0 msgr2=0x7f08840b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.209+0000 7f088aa6a640 1 -- 172.21.15.71:0/2456808093 shutdown_connections 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.209+0000 7f088aa6a640 1 -- 172.21.15.71:0/2456808093 wait complete. 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 10 monclient: build_initial_monmap 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 10 monclient: monmap: 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:22.217 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 1 librados: starting msgr at 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 1 librados: starting objecter 2024-03-29T11:41:22.218 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 1 Processor -- start 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 1 -- start start 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 1 librados: setting wanted keys 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 1 librados: calling monclient init 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.210+0000 7f088aa6a640 10 monclient: init 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient(hunting): picked mon.noname-a con 0x7f08840ad6c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient(hunting): picked mon.noname-c con 0x7f08840af6c0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient(hunting): picked mon.noname-b con 0x7f088414fdc0 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:22.220 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08841eff00 con 0x7f08840ad6c0 2024-03-29T11:41:22.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08841ec8d0 con 0x7f088414fdc0 2024-03-29T11:41:22.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08841f0aa0 con 0x7f08840af6c0 2024-03-29T11:41:22.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:22.221 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.211+0000 7f088aa6a640 10 monclient(hunting): authenticate will time out at 1336.365941s 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.219+0000 7f0889007640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f088414fdc0 0x7f0884140910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:42283/0 (socket says 172.21.15.71:42283) 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.219+0000 7f0889007640 1 -- 172.21.15.71:0/748506123 learned_addr learned my addr 172.21.15.71:0/748506123 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.219+0000 7f08817fa640 10 client.?.objecter ms_handle_connect 0x7f08840ad6c0 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.219+0000 7f08817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.219+0000 7f08817fa640 10 client.?.objecter ms_handle_connect 0x7f088414fdc0 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.219+0000 7f08817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:22.222 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.220+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512612617 0 0) 0x7f08841ec8d0 con 0x7f088414fdc0 2024-03-29T11:41:22.223 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.220+0000 7f08817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.220+0000 7f08817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.220+0000 7f08817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.220+0000 7f08817fa640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.220+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0870004ee0 con 0x7f088414fdc0 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.221+0000 7f08817fa640 10 client.?.objecter ms_handle_connect 0x7f08840af6c0 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.221+0000 7f08817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:22.224 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.221+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426858954 0 0) 0x7f08841f0aa0 con 0x7f08840af6c0 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08841ec8d0 con 0x7f08840af6c0 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625960513 0 0) 0x7f08841eff00 con 0x7f08840ad6c0 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08841f0aa0 con 0x7f08840ad6c0 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 497541629 0 0) 0x7f0870004ee0 con 0x7f088414fdc0 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08841eff00 con 0x7f088414fdc0 2024-03-29T11:41:22.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f0874001890 con 0x7f088414fdc0 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): dump: 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:22.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.222+0000 7f08817fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:22.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617677311 0 0) 0x7f08841ec8d0 con 0x7f08840af6c0 2024-03-29T11:41:22.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0870004ee0 con 0x7f08840af6c0 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2257752812 0 0) 0x7f08841eff00 con 0x7f088414fdc0 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 >> v1:172.21.15.71:6790/0 conn(0x7f08840af6c0 legacy=0x7f0884140200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 >> v1:172.21.15.71:6789/0 conn(0x7f08840ad6c0 legacy=0x7f08841f0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 monclient: found mon.b 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08841f0c30 con 0x7f088414fdc0 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f08817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 10 monclient: _renew_subs 2024-03-29T11:41:22.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088406f840 con 0x7f088414fdc0 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 10 monclient: _renew_subs 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f088414a060 con 0x7f088414fdc0 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.223+0000 7f088aa6a640 1 librados: init done 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.227+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0874003610 con 0x7f088414fdc0 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.227+0000 7f08817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.227+0000 7f0882ffd640 4 set_mon_vals no callback set 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.228+0000 7f088aa6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.228+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f0874001d80 con 0x7f088414fdc0 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.228+0000 7f08817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.228+0000 7f08817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.228+0000 7f08817fa640 10 monclient: dump: 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.229+0000 7f088aa6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.229+0000 7f088aa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:41:22.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.229+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864005080 con 0x7f088414fdc0 2024-03-29T11:41:22.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.230+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f0874003a20 con 0x7f088414fdc0 2024-03-29T11:41:22.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.230+0000 7f0888806640 10 monclient: get_auth_request con 0x7f087006e040 auth_method 0 2024-03-29T11:41:22.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.231+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f08740544d0 con 0x7f088414fdc0 2024-03-29T11:41:22.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.231+0000 7f08817fa640 10 client.4540.objecter ms_dispatch 0x7f0884000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:22.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.231+0000 7f08817fa640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:22.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.231+0000 7f08817fa640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:22.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.231+0000 7f08817fa640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:41:22.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.231+0000 7f08817fa640 10 client.4540.objecter ms_handle_connect 0x7f087006e040 2024-03-29T11:41:22.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.237+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0874050770 con 0x7f088414fdc0 2024-03-29T11:41:22.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.237+0000 7f08817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:22.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.237+0000 7f08817fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:22.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.601+0000 7f088aa6a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:22.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.601+0000 7f088aa6a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:22.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.601+0000 7f088aa6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:41:22.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.601+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0864005330 con 0x7f088414fdc0 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.602+0000 7f08817fa640 1 -- 172.21.15.71:0/748506123 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f0874050770 con 0x7f088414fdc0 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.602+0000 7f08817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.602+0000 7f08817fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 10 librados: watch_flush enter 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 10 librados: watch_flush exit 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 >> v1:172.21.15.71:6816/1596514235 conn(0x7f087006e040 legacy=0x7f0870070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 10 monclient: shutdown 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:22.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 >> v1:172.21.15.81:6789/0 conn(0x7f088414fdc0 legacy=0x7f0884140910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 shutdown_connections 2024-03-29T11:41:22.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 >> 172.21.15.71:0/748506123 conn(0x7f08840b35e0 msgr2=0x7f08840b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:22.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 shutdown_connections 2024-03-29T11:41:22.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.607+0000 7f088aa6a640 1 -- 172.21.15.71:0/748506123 wait complete. 2024-03-29T11:41:22.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:22.608+0000 7f088aa6a640 1 librados: shutdown 2024-03-29T11:41:22.628 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:29.436 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:29.584 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.581+0000 7fb321dbc640 10 monclient: get_monmap_and_config 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.581+0000 7fb321dbc640 10 monclient: build_initial_monmap 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.581+0000 7fb321dbc640 10 monclient: monmap: 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:29.582617+0000 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:29.582617+0000 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:29.585 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:29.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.584+0000 7fb321dbc640 1 Processor -- start 2024-03-29T11:41:29.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.584+0000 7fb321dbc640 1 -- start start 2024-03-29T11:41:29.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.584+0000 7fb321dbc640 10 monclient: init 2024-03-29T11:41:29.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.585+0000 7fb321dbc640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:29.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.585+0000 7fb321dbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:29.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.586+0000 7fb321dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fb314006a90 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:29.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.589+0000 7fb31b7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb314006a90 0x7fb3140c8d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50861/0 (socket says 172.21.15.71:50861) 2024-03-29T11:41:29.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.589+0000 7fb31affd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb31402b120 0x7fb3140cf120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:59053/0 (socket says 172.21.15.71:59053) 2024-03-29T11:41:29.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.589+0000 7fb321dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fb31402b120 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.589+0000 7fb321dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fb314029c50 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.589+0000 7fb321dbc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31402a290 con 0x7fb314029c50 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.589+0000 7fb321dbc640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31402c460 con 0x7fb31402b120 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.590+0000 7fb321dbc640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3140c9460 con 0x7fb314006a90 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.590+0000 7fb321dbc640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.590+0000 7fb321dbc640 10 monclient(hunting): authenticate will time out at 1343.744061s 2024-03-29T11:41:29.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.590+0000 7fb31b7fe640 1 -- 172.21.15.71:0/2818076442 learned_addr learned my addr 172.21.15.71:0/2818076442 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.590+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145764318 0 0) 0x7fb31402c460 con 0x7fb31402b120 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c003520 con 0x7fb31402b120 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939508688 0 0) 0x7fb3140c9460 con 0x7fb314006a90 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:29.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31402c460 con 0x7fb314006a90 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285532604 0 0) 0x7fb30c003520 con 0x7fb31402b120 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 >> v1:172.21.15.71:6790/0 conn(0x7fb314006a90 legacy=0x7fb3140c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 >> v1:172.21.15.71:6789/0 conn(0x7fb314029c50 legacy=0x7fb3140d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140c95f0 con 0x7fb31402b120 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb31c04f730 con 0x7fb31402b120 2024-03-29T11:41:29.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.591+0000 7fb31a7fc640 10 monclient: dump: 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:29.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb321dbc640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb321dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb31c0008d0 con 0x7fb31402b120 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb31a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb31a7fc640 1 -- 172.21.15.71:0/2818076442 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb31c0758b0 con 0x7fb31402b120 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb31a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb31a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb31a7fc640 10 monclient: dump: 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:29.598 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb321dbc640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.592+0000 7fb321dbc640 4 set_mon_vals no callback set 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.594+0000 7fb321dbc640 10 monclient: shutdown 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.594+0000 7fb321dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.594+0000 7fb321dbc640 1 -- 172.21.15.71:0/2818076442 >> v1:172.21.15.81:6789/0 conn(0x7fb31402b120 legacy=0x7fb3140cf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.595+0000 7fb321dbc640 1 -- 172.21.15.71:0/2818076442 shutdown_connections 2024-03-29T11:41:29.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.595+0000 7fb321dbc640 1 -- 172.21.15.71:0/2818076442 >> 172.21.15.71:0/2818076442 conn(0x7fb314005910 msgr2=0x7fb314005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:29.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.596+0000 7fb321dbc640 1 -- 172.21.15.71:0/2818076442 shutdown_connections 2024-03-29T11:41:29.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.596+0000 7fb321dbc640 1 -- 172.21.15.71:0/2818076442 wait complete. 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.596+0000 7fb321dbc640 10 monclient: build_initial_monmap 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.597+0000 7fb321dbc640 10 monclient: monmap: 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:29.602 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.597+0000 7fb321dbc640 1 librados: starting msgr at 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.597+0000 7fb321dbc640 1 librados: starting objecter 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.597+0000 7fb321dbc640 1 Processor -- start 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.598+0000 7fb321dbc640 1 -- start start 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.598+0000 7fb321dbc640 1 librados: setting wanted keys 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.598+0000 7fb321dbc640 1 librados: calling monclient init 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.598+0000 7fb321dbc640 10 monclient: init 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient(hunting): picked mon.noname-b con 0x7fb314006a90 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient(hunting): picked mon.noname-a con 0x7fb314029c50 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient(hunting): picked mon.noname-c con 0x7fb3140c22e0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb314158710 con 0x7fb314029c50 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb314157410 con 0x7fb314006a90 2024-03-29T11:41:29.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3141576f0 con 0x7fb3140c22e0 2024-03-29T11:41:29.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:29.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.599+0000 7fb321dbc640 10 monclient(hunting): authenticate will time out at 1343.753568s 2024-03-29T11:41:29.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.602+0000 7fb31bfff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fb3140c22e0 0x7fb314155090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:50965/0 (socket says 172.21.15.71:50965) 2024-03-29T11:41:29.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb31bfff640 1 -- 172.21.15.71:0/777203979 learned_addr learned my addr 172.21.15.71:0/777203979 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 client.?.objecter ms_handle_connect 0x7fb3140c22e0 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 client.?.objecter ms_handle_connect 0x7fb314029c50 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513748269 0 0) 0x7fb3141576f0 con 0x7fb3140c22e0 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.603+0000 7fb307fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.604+0000 7fb307fff640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:41:29.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.604+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb300004eb0 con 0x7fb3140c22e0 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.604+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368309837 0 0) 0x7fb314158710 con 0x7fb314029c50 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.604+0000 7fb307fff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.604+0000 7fb307fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3141576f0 con 0x7fb314029c50 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250091521 0 0) 0x7fb300004eb0 con 0x7fb3140c22e0 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb314158710 con 0x7fb3140c22e0 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb3080019b0 con 0x7fb3140c22e0 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 10 monclient(hunting): dump: 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:29.611 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.605+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1642976718 0 0) 0x7fb314158710 con 0x7fb3140c22e0 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 >> v1:172.21.15.81:6789/0 conn(0x7fb314006a90 legacy=0x7fb314157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 >> v1:172.21.15.71:6789/0 conn(0x7fb314029c50 legacy=0x7fb3140c1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 1 monclient: found mon.c 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:29.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140c1ca0 con 0x7fb3140c22e0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 10 monclient: _finish_auth 0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb307fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 10 monclient: _renew_subs 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3140c1f50 con 0x7fb3140c22e0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 10 monclient: _renew_subs 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb314155980 con 0x7fb3140c22e0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.606+0000 7fb321dbc640 1 librados: init done 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb308001cb0 con 0x7fb3140c22e0 2024-03-29T11:41:29.613 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb307fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb3197fa640 4 set_mon_vals no callback set 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fb308003a60 con 0x7fb3140c22e0 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb307fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.607+0000 7fb307fff640 10 monclient: dump: 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:29.614 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:29.615 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:29.615 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:29.615 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:29.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.608+0000 7fb321dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:29.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.608+0000 7fb321dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.608+0000 7fb321dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.608+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc005080 con 0x7fb3140c22e0 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.609+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fb308003bc0 con 0x7fb3140c22e0 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7fb308084bc0 con 0x7fb3140c22e0 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb307fff640 10 client.4631.objecter ms_dispatch 0x7fb31c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb307fff640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb307fff640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb307fff640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb31b7fe640 10 monclient: get_auth_request con 0x7fb30006dc50 auth_method 0 2024-03-29T11:41:29.616 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.610+0000 7fb307fff640 10 client.4631.objecter ms_handle_connect 0x7fb30006dc50 2024-03-29T11:41:29.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.617+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3080504a0 con 0x7fb3140c22e0 2024-03-29T11:41:29.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.617+0000 7fb307fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:29.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.617+0000 7fb307fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:29.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.945+0000 7fb321dbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:29.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.945+0000 7fb321dbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:29.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.945+0000 7fb321dbc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:29.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.945+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2fc005330 con 0x7fb3140c22e0 2024-03-29T11:41:29.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.949+0000 7fb307fff640 1 -- 172.21.15.71:0/777203979 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7fb3080523d0 con 0x7fb3140c22e0 2024-03-29T11:41:29.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.949+0000 7fb307fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:29.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.949+0000 7fb307fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:29.956 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:29.956 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.955+0000 7fb321dbc640 10 librados: watch_flush enter 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.955+0000 7fb321dbc640 10 librados: watch_flush exit 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.955+0000 7fb321dbc640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.955+0000 7fb321dbc640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.955+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 >> v1:172.21.15.71:6816/1596514235 conn(0x7fb30006dc50 legacy=0x7fb300070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.955+0000 7fb321dbc640 10 monclient: shutdown 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 >> v1:172.21.15.71:6790/0 conn(0x7fb3140c22e0 legacy=0x7fb314155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:29.959 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 shutdown_connections 2024-03-29T11:41:29.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 >> 172.21.15.71:0/777203979 conn(0x7fb314005910 msgr2=0x7fb314037570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:29.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 shutdown_connections 2024-03-29T11:41:29.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 1 -- 172.21.15.71:0/777203979 wait complete. 2024-03-29T11:41:29.960 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:29.956+0000 7fb321dbc640 1 librados: shutdown 2024-03-29T11:41:29.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:36.783 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.918+0000 7f5b24825640 10 monclient: get_monmap_and_config 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.918+0000 7f5b24825640 10 monclient: build_initial_monmap 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.918+0000 7f5b24825640 10 monclient: monmap: 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:36.919677+0000 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:36.919677+0000 2024-03-29T11:41:36.921 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.920+0000 7f5b24825640 1 Processor -- start 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.920+0000 7f5b24825640 1 -- start start 2024-03-29T11:41:36.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.920+0000 7f5b24825640 10 monclient: init 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient(hunting): picked mon.noname-b con 0x7f5b200b0860 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient(hunting): picked mon.noname-c con 0x7f5b20072450 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient(hunting): picked mon.noname-a con 0x7f5b2013ee40 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b200ad840 con 0x7f5b2013ee40 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2013c830 con 0x7f5b20072450 2024-03-29T11:41:36.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2013c9c0 con 0x7f5b200b0860 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b24825640 10 monclient(hunting): authenticate will time out at 1351.075492s 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1e59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5b200b0860 0x7f5b200acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54186/0 (socket says 172.21.15.71:54186) 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1e59c640 1 -- 172.21.15.71:0/2989470286 learned_addr learned my addr 172.21.15.71:0/2989470286 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904231070 0 0) 0x7f5b200ad840 con 0x7f5b2013ee40 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:36.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b00003520 con 0x7f5b2013ee40 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970910074 0 0) 0x7f5b2013c9c0 con 0x7f5b200b0860 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.921+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b200ad840 con 0x7f5b200b0860 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.922+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4163921651 0 0) 0x7f5b00003520 con 0x7f5b2013ee40 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.922+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 >> v1:172.21.15.71:6790/0 conn(0x7f5b200b0860 legacy=0x7f5b200acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:36.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.922+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 >> v1:172.21.15.81:6789/0 conn(0x7f5b20072450 legacy=0x7f5b2013c120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:36.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:36.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 1 monclient: found mon.noname-a 2024-03-29T11:41:36.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2013f970 con 0x7f5b2013ee40 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5b18001a10 con 0x7f5b2013ee40 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:36.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b1dd9b640 10 monclient: dump: 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:36.931 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b24825640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.923+0000 7f5b24825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b18001bf0 con 0x7f5b2013ee40 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b1dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b24825640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b24825640 4 set_mon_vals no callback set 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b1dd9b640 1 -- 172.21.15.71:0/2989470286 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5b18002000 con 0x7f5b2013ee40 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b1dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b1dd9b640 10 monclient: dump: 2024-03-29T11:41:36.932 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:36.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b24825640 10 monclient: shutdown 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b24825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.924+0000 7f5b24825640 1 -- 172.21.15.71:0/2989470286 >> v1:172.21.15.71:6789/0 conn(0x7f5b2013ee40 legacy=0x7f5b2013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.925+0000 7f5b24825640 1 -- 172.21.15.71:0/2989470286 shutdown_connections 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.925+0000 7f5b24825640 1 -- 172.21.15.71:0/2989470286 >> 172.21.15.71:0/2989470286 conn(0x7f5b2006b660 msgr2=0x7f5b2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.925+0000 7f5b24825640 1 -- 172.21.15.71:0/2989470286 shutdown_connections 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.925+0000 7f5b24825640 1 -- 172.21.15.71:0/2989470286 wait complete. 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.926+0000 7f5b24825640 10 monclient: build_initial_monmap 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.926+0000 7f5b24825640 10 monclient: monmap: 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:36.934 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.927+0000 7f5b24825640 1 librados: starting msgr at 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.927+0000 7f5b24825640 1 librados: starting objecter 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.927+0000 7f5b24825640 1 Processor -- start 2024-03-29T11:41:36.935 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.927+0000 7f5b24825640 1 -- start start 2024-03-29T11:41:36.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.927+0000 7f5b24825640 1 librados: setting wanted keys 2024-03-29T11:41:36.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.928+0000 7f5b24825640 1 librados: calling monclient init 2024-03-29T11:41:36.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.928+0000 7f5b24825640 10 monclient: init 2024-03-29T11:41:36.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.928+0000 7f5b24825640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:36.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.928+0000 7f5b24825640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:36.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.929+0000 7f5b24825640 10 monclient(hunting): picked mon.noname-c con 0x7f5b20072450 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.929+0000 7f5b24825640 10 monclient(hunting): picked mon.noname-a con 0x7f5b200b0860 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.929+0000 7f5b17fff640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f5b200b0860 0x7f5b201d3720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:55180/0 (socket says 172.21.15.71:55180) 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.929+0000 7f5b1e59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f5b20072450 0x7f5b20141ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:55342/0 (socket says 172.21.15.71:55342) 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.929+0000 7f5b17fff640 1 -- 172.21.15.71:0/299028719 learned_addr learned my addr 172.21.15.71:0/299028719 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 client.?.objecter ms_handle_connect 0x7f5b20072450 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 client.?.objecter ms_handle_connect 0x7f5b200b0860 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b24825640 10 monclient(hunting): picked mon.noname-b con 0x7f5b2013ee40 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:36.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b201427b0 con 0x7f5b200b0860 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b20142cb0 con 0x7f5b2013ee40 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2013d4d0 con 0x7f5b20072450 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b24825640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b24825640 10 monclient(hunting): authenticate will time out at 1351.084598s 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748439166 0 0) 0x7f5b201427b0 con 0x7f5b200b0860 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.930+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04004eb0 con 0x7f5b200b0860 2024-03-29T11:41:36.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551494830 0 0) 0x7f5b04004eb0 con 0x7f5b200b0860 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b201427b0 con 0x7f5b200b0860 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5b08001870 con 0x7f5b200b0860 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.950 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085050316 0 0) 0x7f5b2013d4d0 con 0x7f5b20072450 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:36.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.931+0000 7f5b16ffd640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04004eb0 con 0x7f5b20072450 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3544131640 0 0) 0x7f5b201427b0 con 0x7f5b200b0860 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 >> v1:172.21.15.71:6790/0 conn(0x7f5b20072450 legacy=0x7f5b20141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 >> v1:172.21.15.81:6789/0 conn(0x7f5b2013ee40 legacy=0x7f5b201ddd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 1 monclient: found mon.a 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b20142f20 con 0x7f5b200b0860 2024-03-29T11:41:36.952 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 10 monclient: _renew_subs 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b20141160 con 0x7f5b200b0860 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 10 monclient: _renew_subs 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b201d3e30 con 0x7f5b200b0860 2024-03-29T11:41:36.953 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.932+0000 7f5b24825640 1 librados: init done 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.933+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b080019d0 con 0x7f5b200b0860 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.933+0000 7f5b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.933+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5b08004af0 con 0x7f5b200b0860 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.933+0000 7f5b1d59a640 4 set_mon_vals no callback set 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.934+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.934+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.934+0000 7f5b16ffd640 10 monclient: dump: 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:36.954 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.935+0000 7f5b24825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.935+0000 7f5b24825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.935+0000 7f5b24825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:36.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.935+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b200b0c40 con 0x7f5b200b0860 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.935+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f5b0801aac0 con 0x7f5b200b0860 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f5b08084ad0 con 0x7f5b200b0860 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b16ffd640 10 client.4527.objecter ms_dispatch 0x7f5b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b16ffd640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b16ffd640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b16ffd640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b1e59c640 10 monclient: get_auth_request con 0x7f5b0406dc10 auth_method 0 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.936+0000 7f5b16ffd640 10 client.4527.objecter ms_handle_connect 0x7f5b0406dc10 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.944+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b08050410 con 0x7f5b200b0860 2024-03-29T11:41:36.956 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.944+0000 7f5b16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:36.957 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:36.944+0000 7f5b16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.256+0000 7f5b24825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.256+0000 7f5b24825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.256+0000 7f5b24825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:37.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.256+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b20075f10 con 0x7f5b200b0860 2024-03-29T11:41:37.260 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:37.260 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:37.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.257+0000 7f5b16ffd640 1 -- 172.21.15.71:0/299028719 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f5b08052340 con 0x7f5b200b0860 2024-03-29T11:41:37.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.257+0000 7f5b16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:37.260 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.257+0000 7f5b16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:37.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.260+0000 7f5b24825640 10 librados: watch_flush enter 2024-03-29T11:41:37.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.260+0000 7f5b24825640 10 librados: watch_flush exit 2024-03-29T11:41:37.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.260+0000 7f5b24825640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T11:41:37.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.260+0000 7f5b24825640 10 client.4527.objecter successfully canceled tick 2024-03-29T11:41:37.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 >> v1:172.21.15.71:6816/1596514235 conn(0x7f5b0406dc10 legacy=0x7f5b040700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 10 monclient: shutdown 2024-03-29T11:41:37.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:37.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 >> v1:172.21.15.71:6789/0 conn(0x7f5b200b0860 legacy=0x7f5b201d3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 shutdown_connections 2024-03-29T11:41:37.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 >> 172.21.15.71:0/299028719 conn(0x7f5b2006b660 msgr2=0x7f5b20069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:37.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 shutdown_connections 2024-03-29T11:41:37.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 -- 172.21.15.71:0/299028719 wait complete. 2024-03-29T11:41:37.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:37.262+0000 7f5b24825640 1 librados: shutdown 2024-03-29T11:41:37.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:44.086 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.244+0000 7f733ba55640 10 monclient: get_monmap_and_config 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.244+0000 7f733ba55640 10 monclient: build_initial_monmap 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.244+0000 7f733ba55640 10 monclient: monmap: 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:44.245152+0000 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:44.245152+0000 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:44.247 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:44.248 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:44.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.245+0000 7f733ba55640 1 Processor -- start 2024-03-29T11:41:44.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.245+0000 7f733ba55640 1 -- start start 2024-03-29T11:41:44.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.245+0000 7f733ba55640 10 monclient: init 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient(hunting): picked mon.noname-a con 0x7f73340b48a0 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient(hunting): picked mon.noname-c con 0x7f73340b3f70 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient(hunting): picked mon.noname-b con 0x7f733413fb20 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340b43b0 con 0x7f73340b48a0 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340b0b90 con 0x7f73340b3f70 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340b19a0 con 0x7f733413fb20 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:44.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.246+0000 7f733ba55640 10 monclient(hunting): authenticate will time out at 1358.400751s 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f73397f1640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f73340b48a0 0x7f73340b1290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:42004/0 (socket says 172.21.15.71:42004) 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f7339ff2640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f733413fb20 0x7f733414da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34002/0 (socket says 172.21.15.71:34002) 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f7339ff2640 1 -- 172.21.15.71:0/494494447 learned_addr learned my addr 172.21.15.71:0/494494447 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569257850 0 0) 0x7f73340b43b0 con 0x7f73340b48a0 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7320003520 con 0x7f73340b48a0 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390779934 0 0) 0x7f73340b19a0 con 0x7f733413fb20 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.247+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73340b43b0 con 0x7f733413fb20 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570199911 0 0) 0x7f7320003520 con 0x7f73340b48a0 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 >> v1:172.21.15.71:6790/0 conn(0x7f733413fb20 legacy=0x7f733414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 >> v1:172.21.15.81:6789/0 conn(0x7f73340b3f70 legacy=0x7f733413f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 monclient: found mon.noname-a 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73340af800 con 0x7f73340b48a0 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:44.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7330001690 con 0x7f73340b48a0 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: dump: 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:44.255 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f733ba55640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f733ba55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73300017f0 con 0x7f73340b48a0 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 1 -- 172.21.15.71:0/494494447 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f7330001c00 con 0x7f73340b48a0 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f732bfff640 10 monclient: dump: 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:44.256 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f733ba55640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.248+0000 7f733ba55640 4 set_mon_vals no callback set 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.249+0000 7f733ba55640 10 monclient: shutdown 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.249+0000 7f733ba55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:44.257 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.249+0000 7f733ba55640 1 -- 172.21.15.71:0/494494447 >> v1:172.21.15.71:6789/0 conn(0x7f73340b48a0 legacy=0x7f73340b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.249+0000 7f733ba55640 1 -- 172.21.15.71:0/494494447 shutdown_connections 2024-03-29T11:41:44.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.249+0000 7f733ba55640 1 -- 172.21.15.71:0/494494447 >> 172.21.15.71:0/494494447 conn(0x7f733406c8a0 msgr2=0x7f733406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:44.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.249+0000 7f733ba55640 1 -- 172.21.15.71:0/494494447 shutdown_connections 2024-03-29T11:41:44.258 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.250+0000 7f733ba55640 1 -- 172.21.15.71:0/494494447 wait complete. 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.250+0000 7f733ba55640 10 monclient: build_initial_monmap 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.250+0000 7f733ba55640 10 monclient: monmap: 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:44.263 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 1 librados: starting msgr at 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 1 librados: starting objecter 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 1 Processor -- start 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 1 -- start start 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 1 librados: setting wanted keys 2024-03-29T11:41:44.264 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 1 librados: calling monclient init 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.251+0000 7f733ba55640 10 monclient: init 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient(hunting): picked mon.noname-b con 0x7f73340b3f70 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient(hunting): picked mon.noname-c con 0x7f73340b48a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient(hunting): picked mon.noname-a con 0x7f733413fb20 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7334071cd0 con 0x7f733413fb20 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73340712a0 con 0x7f73340b3f70 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7334071f20 con 0x7f73340b48a0 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.252+0000 7f733ba55640 10 monclient(hunting): authenticate will time out at 1358.406866s 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f73397f1640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f73340b3f70 0x7f733406f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:36260/0 (socket says 172.21.15.71:36260) 2024-03-29T11:41:44.265 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f73397f1640 1 -- 172.21.15.71:0/1320723818 learned_addr learned my addr 172.21.15.71:0/1320723818 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 client.?.objecter ms_handle_connect 0x7f73340b3f70 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 client.?.objecter ms_handle_connect 0x7f733413fb20 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497505567 0 0) 0x7f73340712a0 con 0x7f73340b3f70 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7318004eb0 con 0x7f73340b3f70 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 client.?.objecter ms_handle_connect 0x7f73340b48a0 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:44.266 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976621329 0 0) 0x7f7334071cd0 con 0x7f733413fb20 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.253+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73340712a0 con 0x7f733413fb20 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929073076 0 0) 0x7f7334071f20 con 0x7f73340b48a0 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7334071cd0 con 0x7f73340b48a0 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17156092 0 0) 0x7f73340712a0 con 0x7f733413fb20 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7334071f20 con 0x7f733413fb20 2024-03-29T11:41:44.267 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f732c001b50 con 0x7f733413fb20 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:44.268 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 234951778 0 0) 0x7f7334071f20 con 0x7f733413fb20 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 >> v1:172.21.15.71:6790/0 conn(0x7f73340b48a0 legacy=0x7f733406ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 >> v1:172.21.15.81:6789/0 conn(0x7f73340b3f70 legacy=0x7f733406f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 monclient: found mon.a 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7334070010 con 0x7f733413fb20 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.254+0000 7f7329ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 10 monclient: _renew_subs 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:44.269 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733406f350 con 0x7f733413fb20 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 10 monclient: _renew_subs 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f733413b8d0 con 0x7f733413fb20 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f733ba55640 1 librados: init done 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f732c0020c0 con 0x7f733413fb20 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f7329ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f732c004d00 con 0x7f733413fb20 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f7329ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f732b7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f7329ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:44.270 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.255+0000 7f7329ffb640 10 monclient: dump: 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:44.271 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.257+0000 7f733ba55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.257+0000 7f733ba55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.257+0000 7f733ba55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.257+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7314005080 con 0x7f733413fb20 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.259+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f732c01acd0 con 0x7f733413fb20 2024-03-29T11:41:44.272 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.260+0000 7f73397f1640 10 monclient: get_auth_request con 0x7f731806de50 auth_method 0 2024-03-29T11:41:44.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.261+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f732c084be0 con 0x7f733413fb20 2024-03-29T11:41:44.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.261+0000 7f7329ffb640 10 client.4533.objecter ms_dispatch 0x7f7334000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:44.273 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.261+0000 7f7329ffb640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:44.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.261+0000 7f7329ffb640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:44.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.261+0000 7f7329ffb640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:41:44.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.261+0000 7f7329ffb640 10 client.4533.objecter ms_handle_connect 0x7f731806de50 2024-03-29T11:41:44.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.269+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f732c0504c0 con 0x7f733413fb20 2024-03-29T11:41:44.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.269+0000 7f7329ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:44.274 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.269+0000 7f7329ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:44.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.620+0000 7f733ba55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:44.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.620+0000 7f733ba55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:44.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.620+0000 7f733ba55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:41:44.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.620+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7314005330 con 0x7f733413fb20 2024-03-29T11:41:44.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.621+0000 7f7329ffb640 1 -- 172.21.15.71:0/1320723818 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f732c0523f0 con 0x7f733413fb20 2024-03-29T11:41:44.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.621+0000 7f7329ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:44.624 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.621+0000 7f7329ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:44.624 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:44.624 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 10 librados: watch_flush enter 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 10 librados: watch_flush exit 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 >> v1:172.21.15.71:6816/1596514235 conn(0x7f731806de50 legacy=0x7f7318070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 10 monclient: shutdown 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:44.628 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 >> v1:172.21.15.71:6789/0 conn(0x7f733413fb20 legacy=0x7f73341e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 shutdown_connections 2024-03-29T11:41:44.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 >> 172.21.15.71:0/1320723818 conn(0x7f733406c8a0 msgr2=0x7f73340b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:44.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 shutdown_connections 2024-03-29T11:41:44.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.625+0000 7f733ba55640 1 -- 172.21.15.71:0/1320723818 wait complete. 2024-03-29T11:41:44.629 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:44.626+0000 7f733ba55640 1 librados: shutdown 2024-03-29T11:41:44.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:51.449 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:51.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.592+0000 7f00df32e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:51.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.592+0000 7f00df32e640 10 monclient: build_initial_monmap 2024-03-29T11:41:51.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.592+0000 7f00df32e640 10 monclient: monmap: 2024-03-29T11:41:51.596 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:51.596 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:51.593676+0000 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:51.593676+0000 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.593+0000 7f00df32e640 1 Processor -- start 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.593+0000 7f00df32e640 1 -- start start 2024-03-29T11:41:51.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.593+0000 7f00df32e640 10 monclient: init 2024-03-29T11:41:51.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:51.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:51.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient(hunting): picked mon.noname-b con 0x7f00d80b3f30 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient(hunting): picked mon.noname-c con 0x7f00d80a9750 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient(hunting): picked mon.noname-a con 0x7f00d813b9d0 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d8142490 con 0x7f00d813b9d0 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d80ac130 con 0x7f00d80a9750 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d80ac410 con 0x7f00d80b3f30 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.595+0000 7f00df32e640 10 monclient(hunting): authenticate will time out at 1365.749184s 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.596+0000 7f00de32c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f00d80b3f30 0x7f00d8141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:51500/0 (socket says 172.21.15.71:51500) 2024-03-29T11:41:51.599 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.596+0000 7f00de32c640 1 -- 172.21.15.71:0/34177399 learned_addr learned my addr 172.21.15.71:0/34177399 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:51.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.598+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214942651 0 0) 0x7f00d80ac410 con 0x7f00d80b3f30 2024-03-29T11:41:51.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.598+0000 7f00dd32a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:51.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.598+0000 7f00dd32a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:51.600 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.598+0000 7f00dd32a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:51.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.598+0000 7f00dd32a640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:41:51.601 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.598+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c4003520 con 0x7f00d80b3f30 2024-03-29T11:41:51.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.599+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834128473 0 0) 0x7f00d80ac130 con 0x7f00d80a9750 2024-03-29T11:41:51.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.599+0000 7f00dd32a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:51.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.599+0000 7f00dd32a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:51.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.599+0000 7f00dd32a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:51.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.599+0000 7f00dd32a640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:41:51.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.599+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d80ac410 con 0x7f00d80a9750 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920032870 0 0) 0x7f00c4003520 con 0x7f00d80b3f30 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 >> v1:172.21.15.81:6789/0 conn(0x7f00d80a9750 legacy=0x7f00d813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 >> v1:172.21.15.71:6789/0 conn(0x7f00d813b9d0 legacy=0x7f00d81428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 1 monclient: found mon.noname-b 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d80afa40 con 0x7f00d80b3f30 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 10 monclient: _finish_auth 0 2024-03-29T11:41:51.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.600+0000 7f00dd32a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00df32e640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00df32e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f00c8001690 con 0x7f00d80b3f30 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00dd32a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00dd32a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00dd32a640 10 monclient: dump: 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:51.606 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00df32e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00c80018b0 con 0x7f00d80b3f30 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00dd32a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00df32e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.601+0000 7f00df32e640 4 set_mon_vals no callback set 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.602+0000 7f00dd32a640 1 -- 172.21.15.71:0/34177399 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f00c8001c40 con 0x7f00d80b3f30 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.602+0000 7f00dd32a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:51.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.602+0000 7f00dd32a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:51.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.602+0000 7f00df32e640 10 monclient: shutdown 2024-03-29T11:41:51.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.603+0000 7f00dd32a640 10 monclient: dump: 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.603+0000 7f00df32e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.603+0000 7f00df32e640 1 -- 172.21.15.71:0/34177399 >> v1:172.21.15.71:6790/0 conn(0x7f00d80b3f30 legacy=0x7f00d8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.604+0000 7f00df32e640 1 -- 172.21.15.71:0/34177399 shutdown_connections 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.604+0000 7f00df32e640 1 -- 172.21.15.71:0/34177399 >> 172.21.15.71:0/34177399 conn(0x7f00d80673e0 msgr2=0x7f00d80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:51.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.604+0000 7f00df32e640 1 -- 172.21.15.71:0/34177399 shutdown_connections 2024-03-29T11:41:51.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.604+0000 7f00df32e640 1 -- 172.21.15.71:0/34177399 wait complete. 2024-03-29T11:41:51.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.605+0000 7f00df32e640 10 monclient: build_initial_monmap 2024-03-29T11:41:51.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.605+0000 7f00df32e640 10 monclient: monmap: 2024-03-29T11:41:51.610 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:51.610 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.606+0000 7f00df32e640 1 librados: starting msgr at 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.606+0000 7f00df32e640 1 librados: starting objecter 2024-03-29T11:41:51.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.606+0000 7f00df32e640 1 Processor -- start 2024-03-29T11:41:51.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.609+0000 7f00df32e640 1 -- start start 2024-03-29T11:41:51.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.609+0000 7f00df32e640 1 librados: setting wanted keys 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.609+0000 7f00df32e640 1 librados: calling monclient init 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.609+0000 7f00df32e640 10 monclient: init 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient(hunting): picked mon.noname-a con 0x7f00d80a9750 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient(hunting): picked mon.noname-b con 0x7f00d80b3f30 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient(hunting): picked mon.noname-c con 0x7f00d813b9d0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d81355c0 con 0x7f00d80a9750 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d813dc60 con 0x7f00d80b3f30 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00d813cad0 con 0x7f00d813b9d0 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.610+0000 7f00df32e640 10 monclient(hunting): authenticate will time out at 1365.764295s 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.611+0000 7f00deb2d640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f00d813b9d0 0x7f00d8134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56788/0 (socket says 172.21.15.71:56788) 2024-03-29T11:41:51.619 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.611+0000 7f00deb2d640 1 -- 172.21.15.71:0/534945431 learned_addr learned my addr 172.21.15.71:0/534945431 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 client.?.objecter ms_handle_connect 0x7f00d813b9d0 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189741916 0 0) 0x7f00d813cad0 con 0x7f00d813b9d0 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc004eb0 con 0x7f00d813b9d0 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2290764211 0 0) 0x7f00bc004eb0 con 0x7f00d813b9d0 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00d813cad0 con 0x7f00d813b9d0 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f00d0001ad0 con 0x7f00d813b9d0 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:51.620 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): dump: 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1525425292 0 0) 0x7f00d813cad0 con 0x7f00d813b9d0 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.612+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 >> v1:172.21.15.81:6789/0 conn(0x7f00d80b3f30 legacy=0x7f00d81347a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:51.621 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.613+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 >> v1:172.21.15.71:6789/0 conn(0x7f00d80a9750 legacy=0x7f00d814e470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 1 monclient: found mon.c 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d813c1a0 con 0x7f00d813b9d0 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 10 monclient: _renew_subs 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:51.622 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d813be30 con 0x7f00d813b9d0 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 10 monclient: _renew_subs 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d814ed60 con 0x7f00d813b9d0 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.614+0000 7f00df32e640 1 librados: init done 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.615+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00d00020a0 con 0x7f00d813b9d0 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.615+0000 7f00ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.615+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f00d0004d80 con 0x7f00d813b9d0 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.615+0000 7f00ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.615+0000 7f00ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.615+0000 7f00ceffd640 10 monclient: dump: 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:51.623 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:51.624 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:51.624 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:51.624 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:51.624 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:51.624 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:51.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.616+0000 7f00dcb29640 4 set_mon_vals no callback set 2024-03-29T11:41:51.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.618+0000 7f00df32e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:51.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.618+0000 7f00df32e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:51.625 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.618+0000 7f00df32e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.618+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c0005080 con 0x7f00d813b9d0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.619+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f00d0004ee0 con 0x7f00d813b9d0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.620+0000 7f00de32c640 10 monclient: get_auth_request con 0x7f00bc06dcb0 auth_method 0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.620+0000 7f00ceffd640 10 client.4649.objecter ms_handle_connect 0x7f00bc06dcb0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.621+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4109080917 0 0) 0x7f00d0084df0 con 0x7f00d813b9d0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.621+0000 7f00ceffd640 10 client.4649.objecter ms_dispatch 0x7f00d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.622+0000 7f00ceffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.622+0000 7f00ceffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:51.626 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.622+0000 7f00ceffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-29T11:41:51.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.630+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00d0050500 con 0x7f00d813b9d0 2024-03-29T11:41:51.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.630+0000 7f00ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:51.632 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.630+0000 7f00ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:51.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.947+0000 7f00df32e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:51.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.947+0000 7f00df32e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:51.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.947+0000 7f00df32e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:51.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.947+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00c0005330 con 0x7f00d813b9d0 2024-03-29T11:41:51.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.948+0000 7f00ceffd640 1 -- 172.21.15.71:0/534945431 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3796728259) 0x7f00d0052600 con 0x7f00d813b9d0 2024-03-29T11:41:51.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.948+0000 7f00ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:51.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.948+0000 7f00ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:51.951 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:51.951 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":10,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:31:55.500437+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:31:55.500437+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:51.954 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 10 librados: watch_flush enter 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 10 librados: watch_flush exit 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 10 client.4649.objecter successfully canceled tick 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 >> v1:172.21.15.71:6816/1596514235 conn(0x7f00bc06dcb0 legacy=0x7f00bc070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 10 monclient: shutdown 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.951+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 >> v1:172.21.15.71:6790/0 conn(0x7f00d813b9d0 legacy=0x7f00d8134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.952+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 shutdown_connections 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.952+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 >> 172.21.15.71:0/534945431 conn(0x7f00d80673e0 msgr2=0x7f00d813b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.952+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 shutdown_connections 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.952+0000 7f00df32e640 1 -- 172.21.15.71:0/534945431 wait complete. 2024-03-29T11:41:51.955 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:51.952+0000 7f00df32e640 1 librados: shutdown 2024-03-29T11:41:51.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:58.775 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:58.909 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.907+0000 7f90d70d9640 10 monclient: get_monmap_and_config 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.907+0000 7f90d70d9640 10 monclient: build_initial_monmap 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.907+0000 7f90d70d9640 10 monclient: monmap: 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:41:58.908194+0000 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:41:58.908194+0000 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:41:58.910 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:41:58.911 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.908+0000 7f90d70d9640 1 Processor -- start 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.908+0000 7f90d70d9640 1 -- start start 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.908+0000 7f90d70d9640 10 monclient: init 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.908+0000 7f90d70d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.908+0000 7f90d70d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 10 monclient(hunting): picked mon.noname-b con 0x7f90d01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 10 monclient(hunting): picked mon.noname-c con 0x7f90d013f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 10 monclient(hunting): picked mon.noname-a con 0x7f90d00a9d70 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d00aaef0 con 0x7f90d00a9d70 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0145bd0 con 0x7f90d013f650 2024-03-29T11:41:58.913 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0145eb0 con 0x7f90d01457f0 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d70d9640 10 monclient(hunting): authenticate will time out at 1373.063248s 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d4e75640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f90d01457f0 0x7f90d013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:56530/0 (socket says 172.21.15.71:56530) 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90d4e75640 1 -- 172.21.15.71:0/3331479359 learned_addr learned my addr 172.21.15.71:0/3331479359 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265459482 0 0) 0x7f90d0145eb0 con 0x7f90d01457f0 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90cf7fe640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.909+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c0003520 con 0x7f90d01457f0 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.910+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495459384 0 0) 0x7f90c0003520 con 0x7f90d01457f0 2024-03-29T11:41:58.914 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.910+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 >> v1:172.21.15.81:6789/0 conn(0x7f90d013f650 legacy=0x7f90d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:58.915 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.910+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 >> v1:172.21.15.71:6789/0 conn(0x7f90d00a9d70 legacy=0x7f90d014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d0146340 con 0x7f90d01457f0 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:58.917 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f90c4001690 con 0x7f90d01457f0 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: dump: 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:58.918 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90d70d9640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90d70d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c4001860 con 0x7f90d01457f0 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 1 -- 172.21.15.71:0/3331479359 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f90c4001cb0 con 0x7f90d01457f0 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90cf7fe640 10 monclient: dump: 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:58.919 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90d70d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.912+0000 7f90d70d9640 4 set_mon_vals no callback set 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.913+0000 7f90d70d9640 10 monclient: shutdown 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.913+0000 7f90d70d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.913+0000 7f90d70d9640 1 -- 172.21.15.71:0/3331479359 >> v1:172.21.15.71:6790/0 conn(0x7f90d01457f0 legacy=0x7f90d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.914+0000 7f90d70d9640 1 -- 172.21.15.71:0/3331479359 shutdown_connections 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.914+0000 7f90d70d9640 1 -- 172.21.15.71:0/3331479359 >> 172.21.15.71:0/3331479359 conn(0x7f90d0066a00 msgr2=0x7f90d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.914+0000 7f90d70d9640 1 -- 172.21.15.71:0/3331479359 shutdown_connections 2024-03-29T11:41:58.920 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.914+0000 7f90d70d9640 1 -- 172.21.15.71:0/3331479359 wait complete. 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.915+0000 7f90d70d9640 10 monclient: build_initial_monmap 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.915+0000 7f90d70d9640 10 monclient: monmap: 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:41:58.924 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.916+0000 7f90d70d9640 1 librados: starting msgr at 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.916+0000 7f90d70d9640 1 librados: starting objecter 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.916+0000 7f90d70d9640 1 Processor -- start 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.916+0000 7f90d70d9640 1 -- start start 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.916+0000 7f90d70d9640 1 librados: setting wanted keys 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.917+0000 7f90d70d9640 1 librados: calling monclient init 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.917+0000 7f90d70d9640 10 monclient: init 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 10 monclient(hunting): picked mon.noname-b con 0x7f90d00a9d70 addr v1:172.21.15.81:6789/0 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 10 monclient(hunting): picked mon.noname-a con 0x7f90d013f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d5676640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f90d01457f0 0x7f90d00b08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34154/0 (socket says 172.21.15.71:34154) 2024-03-29T11:41:58.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d4e75640 1 --1- 172.21.15.71:0/2001641871 >> v1:172.21.15.81:6789/0 conn(0x7f90d00a9d70 0x7f90d013b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:45396/0 (socket says 172.21.15.71:45396) 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d5676640 1 -- 172.21.15.71:0/2001641871 learned_addr learned my addr 172.21.15.71:0/2001641871 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90cd7fa640 10 client.?.objecter ms_handle_connect 0x7f90d00a9d70 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90cd7fa640 10 client.?.objecter ms_handle_connect 0x7f90d01457f0 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 10 monclient(hunting): picked mon.noname-c con 0x7f90d01457f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d013b970 con 0x7f90d013f650 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0139af0 con 0x7f90d00a9d70 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.919+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d0139c80 con 0x7f90d01457f0 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90d70d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90d70d9640 10 monclient(hunting): authenticate will time out at 1373.074067s 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241218427 0 0) 0x7f90d0139c80 con 0x7f90d01457f0 2024-03-29T11:41:58.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4004eb0 con 0x7f90d01457f0 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3492965214 0 0) 0x7f90b4004eb0 con 0x7f90d01457f0 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d0139c80 con 0x7f90d01457f0 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f90c8001c60 con 0x7f90d01457f0 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): dump: 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.927 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128628735 0 0) 0x7f90d0139af0 con 0x7f90d00a9d70 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:58.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:41:58.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4004eb0 con 0x7f90d00a9d70 2024-03-29T11:41:58.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.920+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3075083214 0 0) 0x7f90d0139c80 con 0x7f90d01457f0 2024-03-29T11:41:58.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.921+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 >> v1:172.21.15.81:6789/0 conn(0x7f90d00a9d70 legacy=0x7f90d013b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:58.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.921+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 >> v1:172.21.15.71:6789/0 conn(0x7f90d013f650 legacy=0x7f90d01393e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.923+0000 7f90cd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.923+0000 7f90cd7fa640 1 monclient: found mon.c 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d0138690 con 0x7f90d01457f0 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:58.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90d70d9640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c8003970 con 0x7f90d01457f0 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90d70d9640 10 monclient: _renew_subs 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90d70d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d0138940 con 0x7f90d01457f0 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f90c8001dc0 con 0x7f90d01457f0 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90cd7fa640 10 monclient: dump: 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.932 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.924+0000 7f90ceffd640 4 set_mon_vals no callback set 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.925+0000 7f90d70d9640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.925+0000 7f90d70d9640 10 monclient: _renew_subs 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.925+0000 7f90d70d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.925+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d0142540 con 0x7f90d01457f0 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.925+0000 7f90d70d9640 1 librados: init done 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.926+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f90c8003d40 con 0x7f90d01457f0 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90d70d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:58.933 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90d70d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90d70d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d0075f10 con 0x7f90d01457f0 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f90c8084db0 con 0x7f90d01457f0 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90cd7fa640 10 client.4655.objecter ms_dispatch 0x7f90d0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90cd7fa640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90cd7fa640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90d4e75640 10 monclient: get_auth_request con 0x7f90b406dd30 auth_method 0 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90cd7fa640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:41:58.934 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.927+0000 7f90cd7fa640 10 client.4655.objecter ms_handle_connect 0x7f90b406dd30 2024-03-29T11:41:58.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.933+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90c80506f0 con 0x7f90d01457f0 2024-03-29T11:41:58.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.933+0000 7f90cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:58.936 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:58.933+0000 7f90cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:59.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.244+0000 7f90d70d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:59.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.244+0000 7f90d70d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:59.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.244+0000 7f90d70d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:41:59.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.244+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90d00b3680 con 0x7f90d01457f0 2024-03-29T11:41:59.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.246+0000 7f90cd7fa640 1 -- 172.21.15.71:0/2001641871 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f90c8052620 con 0x7f90d01457f0 2024-03-29T11:41:59.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.246+0000 7f90cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:59.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.246+0000 7f90cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:59.249 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:41:59.249 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.249+0000 7f90d70d9640 10 librados: watch_flush enter 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.249+0000 7f90d70d9640 10 librados: watch_flush exit 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.250+0000 7f90d70d9640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.250+0000 7f90d70d9640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.250+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 >> v1:172.21.15.71:6816/1596514235 conn(0x7f90b406dd30 legacy=0x7f90b40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.250+0000 7f90d70d9640 10 monclient: shutdown 2024-03-29T11:41:59.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.250+0000 7f90d70d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:59.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.250+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 >> v1:172.21.15.71:6790/0 conn(0x7f90d01457f0 legacy=0x7f90d00b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.251+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 shutdown_connections 2024-03-29T11:41:59.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.251+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 >> 172.21.15.71:0/2001641871 conn(0x7f90d0066a00 msgr2=0x7f90d00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:59.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.251+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 shutdown_connections 2024-03-29T11:41:59.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.251+0000 7f90d70d9640 1 -- 172.21.15.71:0/2001641871 wait complete. 2024-03-29T11:41:59.255 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:41:59.251+0000 7f90d70d9640 1 librados: shutdown 2024-03-29T11:41:59.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:06.075 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:06.226 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.223+0000 7f2d263c8640 10 monclient: get_monmap_and_config 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.223+0000 7f2d263c8640 10 monclient: build_initial_monmap 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.224+0000 7f2d263c8640 10 monclient: monmap: 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:06.224935+0000 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:06.224935+0000 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:06.227 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:06.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.225+0000 7f2d263c8640 1 Processor -- start 2024-03-29T11:42:06.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.225+0000 7f2d263c8640 1 -- start start 2024-03-29T11:42:06.228 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.225+0000 7f2d263c8640 10 monclient: init 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2d20071de0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2d200721c0 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2d20145ae0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d20145650 con 0x7f2d200721c0 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d200ae260 con 0x7f2d20145ae0 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d201420d0 con 0x7f2d20071de0 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:06.229 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.226+0000 7f2d263c8640 10 monclient(hunting): authenticate will time out at 1380.380386s 2024-03-29T11:42:06.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.228+0000 7f2d1ffff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2d20071de0 0x7f2d200aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:46482/0 (socket says 172.21.15.71:46482) 2024-03-29T11:42:06.231 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.228+0000 7f2d1ffff640 1 -- 172.21.15.71:0/530131850 learned_addr learned my addr 172.21.15.71:0/530131850 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.229+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614282018 0 0) 0x7f2d201420d0 con 0x7f2d20071de0 2024-03-29T11:42:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.229+0000 7f2d1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.229+0000 7f2d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.229+0000 7f2d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.229+0000 7f2d1effd640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:42:06.232 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.229+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d08003520 con 0x7f2d20071de0 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 683242729 0 0) 0x7f2d08003520 con 0x7f2d20071de0 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 >> v1:172.21.15.81:6789/0 conn(0x7f2d20145ae0 legacy=0x7f2d20145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 >> v1:172.21.15.71:6789/0 conn(0x7f2d200721c0 legacy=0x7f2d2013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 monclient: found mon.noname-b 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d200b0590 con 0x7f2d20071de0 2024-03-29T11:42:06.233 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: _finish_auth 0 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2d10001690 con 0x7f2d20071de0 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: dump: 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:06.234 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d100017f0 con 0x7f2d20071de0 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 1 -- 172.21.15.71:0/530131850 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2d10001c40 con 0x7f2d20071de0 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.230+0000 7f2d1effd640 10 monclient: dump: 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.235 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.232+0000 7f2d263c8640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.232+0000 7f2d263c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:06.236 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.232+0000 7f2d263c8640 4 set_mon_vals no callback set 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.233+0000 7f2d263c8640 10 monclient: shutdown 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.233+0000 7f2d263c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.233+0000 7f2d263c8640 1 -- 172.21.15.71:0/530131850 >> v1:172.21.15.71:6790/0 conn(0x7f2d20071de0 legacy=0x7f2d200aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.234+0000 7f2d263c8640 1 -- 172.21.15.71:0/530131850 shutdown_connections 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.234+0000 7f2d263c8640 1 -- 172.21.15.71:0/530131850 >> 172.21.15.71:0/530131850 conn(0x7f2d2006b660 msgr2=0x7f2d2006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.235+0000 7f2d263c8640 1 -- 172.21.15.71:0/530131850 shutdown_connections 2024-03-29T11:42:06.237 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.235+0000 7f2d263c8640 1 -- 172.21.15.71:0/530131850 wait complete. 2024-03-29T11:42:06.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.236+0000 7f2d263c8640 10 monclient: build_initial_monmap 2024-03-29T11:42:06.239 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.236+0000 7f2d263c8640 10 monclient: monmap: 2024-03-29T11:42:06.239 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:06.239 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:06.239 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.236+0000 7f2d263c8640 1 librados: starting msgr at 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.236+0000 7f2d263c8640 1 librados: starting objecter 2024-03-29T11:42:06.240 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.236+0000 7f2d263c8640 1 Processor -- start 2024-03-29T11:42:06.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.237+0000 7f2d263c8640 1 -- start start 2024-03-29T11:42:06.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.237+0000 7f2d263c8640 1 librados: setting wanted keys 2024-03-29T11:42:06.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.237+0000 7f2d263c8640 1 librados: calling monclient init 2024-03-29T11:42:06.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.237+0000 7f2d263c8640 10 monclient: init 2024-03-29T11:42:06.241 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.237+0000 7f2d263c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:06.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.238+0000 7f2d263c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:06.242 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 10 monclient(hunting): picked mon.noname-a con 0x7f2d20071de0 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 10 monclient(hunting): picked mon.noname-c con 0x7f2d20145ae0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 10 monclient(hunting): picked mon.noname-b con 0x7f2d2006edd0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d20070b20 con 0x7f2d20071de0 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d20070d90 con 0x7f2d2006edd0 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d20144b70 con 0x7f2d20145ae0 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:06.243 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.240+0000 7f2d263c8640 10 monclient(hunting): authenticate will time out at 1380.394365s 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1f7fe640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f2d20145ae0 0x7f2d2006e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:49082/0 (socket says 172.21.15.71:49082) 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1f7fe640 1 -- 172.21.15.71:0/1648625138 learned_addr learned my addr 172.21.15.71:0/1648625138 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 10 client.?.objecter ms_handle_connect 0x7f2d20145ae0 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164441042 0 0) 0x7f2d20144b70 con 0x7f2d20145ae0 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:06.244 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:42:06.245 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.241+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d0c004eb0 con 0x7f2d20145ae0 2024-03-29T11:42:06.247 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310663125 0 0) 0x7f2d0c004eb0 con 0x7f2d20145ae0 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d20144b70 con 0x7f2d20145ae0 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2d14001a10 con 0x7f2d20145ae0 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 10 monclient(hunting): dump: 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:06.248 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 454041938 0 0) 0x7f2d20144b70 con 0x7f2d20145ae0 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.242+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 >> v1:172.21.15.81:6789/0 conn(0x7f2d2006edd0 legacy=0x7f2d201e6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.243+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 >> v1:172.21.15.71:6789/0 conn(0x7f2d20071de0 legacy=0x7f2d20071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.243+0000 7f2d1cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.243+0000 7f2d1cff9640 1 monclient: found mon.c 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.243+0000 7f2d1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.243+0000 7f2d1cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.243+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d201360b0 con 0x7f2d20145ae0 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d1cff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d1cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:06.249 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 10 monclient: _renew_subs 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2006f260 con 0x7f2d20145ae0 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 10 monclient: _renew_subs 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d201e7a30 con 0x7f2d20145ae0 2024-03-29T11:42:06.250 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.244+0000 7f2d263c8640 1 librados: init done 2024-03-29T11:42:06.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d14001c70 con 0x7f2d20145ae0 2024-03-29T11:42:06.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:06.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f2d14003b20 con 0x7f2d20145ae0 2024-03-29T11:42:06.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.252 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d1cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d1cff9640 10 monclient: dump: 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d263c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d263c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d263c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:06.253 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.246+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d20075f10 con 0x7f2d20145ae0 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.247+0000 7f2d1dffb640 4 set_mon_vals no callback set 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.248+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f2d14003c80 con 0x7f2d20145ae0 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.248+0000 7f2d1ffff640 10 monclient: get_auth_request con 0x7f2d0c06db00 auth_method 0 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.249+0000 7f2d1cff9640 10 client.4661.objecter ms_handle_connect 0x7f2d0c06db00 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.249+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f2d14054a40 con 0x7f2d20145ae0 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.249+0000 7f2d1cff9640 10 client.4661.objecter ms_dispatch 0x7f2d20000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.249+0000 7f2d1cff9640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.249+0000 7f2d1cff9640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:06.254 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.249+0000 7f2d1cff9640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:42:06.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.255+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d14050ce0 con 0x7f2d20145ae0 2024-03-29T11:42:06.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.256+0000 7f2d1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:06.259 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.256+0000 7f2d1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:06.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.592+0000 7f2d263c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:06.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.592+0000 7f2d263c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:06.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.592+0000 7f2d263c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:06.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.592+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d200b2a90 con 0x7f2d20145ae0 2024-03-29T11:42:06.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.594+0000 7f2d1cff9640 1 -- 172.21.15.71:0/1648625138 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f2d14050e80 con 0x7f2d20145ae0 2024-03-29T11:42:06.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.594+0000 7f2d1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:06.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.594+0000 7f2d1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:06.598 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:06.598 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:06.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 10 librados: watch_flush enter 2024-03-29T11:42:06.602 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 10 librados: watch_flush exit 2024-03-29T11:42:06.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:42:06.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:42:06.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 >> v1:172.21.15.71:6816/1596514235 conn(0x7f2d0c06db00 legacy=0x7f2d0c06ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 10 monclient: shutdown 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 >> v1:172.21.15.71:6790/0 conn(0x7f2d20145ae0 legacy=0x7f2d2006e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 shutdown_connections 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.600+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 >> 172.21.15.71:0/1648625138 conn(0x7f2d2006b660 msgr2=0x7f2d200b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.601+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 shutdown_connections 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.601+0000 7f2d263c8640 1 -- 172.21.15.71:0/1648625138 wait complete. 2024-03-29T11:42:06.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:06.601+0000 7f2d263c8640 1 librados: shutdown 2024-03-29T11:42:06.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:13.428 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:13.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.575+0000 7f545f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:42:13.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.575+0000 7f545f59e640 10 monclient: build_initial_monmap 2024-03-29T11:42:13.578 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.575+0000 7f545f59e640 10 monclient: monmap: 2024-03-29T11:42:13.578 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:13.576590+0000 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:13.576590+0000 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:13.579 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:13.580 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.576+0000 7f545f59e640 1 Processor -- start 2024-03-29T11:42:13.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.576+0000 7f545f59e640 1 -- start start 2024-03-29T11:42:13.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.576+0000 7f545f59e640 10 monclient: init 2024-03-29T11:42:13.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.577+0000 7f545f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:13.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.577+0000 7f545f59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:13.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.577+0000 7f545f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f54600b4920 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:13.581 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.577+0000 7f545f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f54600b4200 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:13.586 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.579+0000 7f545e59c640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f54600b4920 0x7f546013f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:54132/0 (socket says 172.21.15.71:54132) 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.580+0000 7f545e59c640 1 -- 172.21.15.71:0/597905478 learned_addr learned my addr 172.21.15.71:0/597905478 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.580+0000 7f545f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f546013f9a0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.580+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600b0930 con 0x7f54600b4200 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.580+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600af460 con 0x7f546013f9a0 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600aeda0 con 0x7f54600b4920 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545f59e640 10 monclient(hunting): authenticate will time out at 1387.735117s 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739877972 0 0) 0x7f54600aeda0 con 0x7f54600b4920 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.587 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5448003520 con 0x7f54600b4920 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 65999729 0 0) 0x7f5448003520 con 0x7f54600b4920 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.581+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 >> v1:172.21.15.81:6789/0 conn(0x7f546013f9a0 legacy=0x7f546014d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 >> v1:172.21.15.71:6789/0 conn(0x7f54600b4200 legacy=0x7f54600b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 1 monclient: found mon.noname-b 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:13.588 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54600af980 con 0x7f54600b4920 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: _finish_auth 0 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5450001690 con 0x7f54600b4920 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: dump: 2024-03-29T11:42:13.589 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:13.590 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545f59e640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54500017f0 con 0x7f54600b4920 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 1 -- 172.21.15.71:0/597905478 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5450001c40 con 0x7f54600b4920 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545f59e640 4 set_mon_vals no callback set 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.591 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.582+0000 7f545d59a640 10 monclient: dump: 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:13.592 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:13.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.583+0000 7f545f59e640 10 monclient: shutdown 2024-03-29T11:42:13.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.583+0000 7f545f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:13.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.583+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 >> v1:172.21.15.71:6790/0 conn(0x7f54600b4920 legacy=0x7f546013f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.583+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 shutdown_connections 2024-03-29T11:42:13.593 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.583+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 >> 172.21.15.71:0/597905478 conn(0x7f546006d230 msgr2=0x7f5460066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:13.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.586+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 shutdown_connections 2024-03-29T11:42:13.594 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.588+0000 7f545f59e640 1 -- 172.21.15.71:0/597905478 wait complete. 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.589+0000 7f545f59e640 10 monclient: build_initial_monmap 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.590+0000 7f545f59e640 10 monclient: monmap: 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.590+0000 7f545f59e640 1 librados: starting msgr at 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.590+0000 7f545f59e640 1 librados: starting objecter 2024-03-29T11:42:13.595 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.591+0000 7f545f59e640 1 Processor -- start 2024-03-29T11:42:13.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.591+0000 7f545f59e640 1 -- start start 2024-03-29T11:42:13.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.591+0000 7f545f59e640 1 librados: setting wanted keys 2024-03-29T11:42:13.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.591+0000 7f545f59e640 1 librados: calling monclient init 2024-03-29T11:42:13.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.591+0000 7f545f59e640 10 monclient: init 2024-03-29T11:42:13.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.592+0000 7f545f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:13.596 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.592+0000 7f545f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:13.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.594+0000 7f545f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f54600b4200 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:13.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.594+0000 7f545e59c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f54600b4200 0x7f54600adbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:59019/0 (socket says 172.21.15.71:59019) 2024-03-29T11:42:13.597 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.594+0000 7f545e59c640 1 -- 172.21.15.71:0/2270165313 learned_addr learned my addr 172.21.15.71:0/2270165313 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:13.598 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f54600b4920 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:13.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f546013f9a0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:13.603 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600ad6b0 con 0x7f54600b4200 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600ad2d0 con 0x7f546013f9a0 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54600ad460 con 0x7f54600b4920 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f545f59e640 10 monclient(hunting): authenticate will time out at 1387.749575s 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f544effd640 10 client.?.objecter ms_handle_connect 0x7f54600b4200 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.595+0000 7f544effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106876552 0 0) 0x7f54600ad6b0 con 0x7f54600b4200 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:13.604 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5440004eb0 con 0x7f54600b4200 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729908187 0 0) 0x7f5440004eb0 con 0x7f54600b4200 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54600ad6b0 con 0x7f54600b4200 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5450001b00 con 0x7f54600b4200 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): dump: 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:13.605 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.596+0000 7f544effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.598+0000 7f544effd640 10 client.?.objecter ms_handle_connect 0x7f54600b4920 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.598+0000 7f544effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:13.606 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.598+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2735188665 0 0) 0x7f54600ad6b0 con 0x7f54600b4200 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.598+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 >> v1:172.21.15.71:6790/0 conn(0x7f54600b4920 legacy=0x7f5460145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.598+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 >> v1:172.21.15.81:6789/0 conn(0x7f546013f9a0 legacy=0x7f5460145da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 1 monclient: found mon.a 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546013fe60 con 0x7f54600b4200 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 10 monclient: _finish_auth 0 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f544effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 10 monclient: _renew_subs 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:13.607 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54601db0e0 con 0x7f54600b4200 2024-03-29T11:42:13.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:13.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 10 monclient: _renew_subs 2024-03-29T11:42:13.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:13.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54601da850 con 0x7f54600b4200 2024-03-29T11:42:13.608 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.599+0000 7f545f59e640 1 librados: init done 2024-03-29T11:42:13.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:13.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:13.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:13.609 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5460140050 con 0x7f54600b4200 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5450001c60 con 0x7f54600b4200 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f5450004d90 con 0x7f54600b4200 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f544effd640 10 monclient: dump: 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:13.610 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.603+0000 7f545cd99640 4 set_mon_vals no callback set 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.605+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f545001ada0 con 0x7f54600b4200 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.605+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f5450084c60 con 0x7f54600b4200 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.605+0000 7f544effd640 10 client.4542.objecter ms_dispatch 0x7f5460000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.605+0000 7f544effd640 3 client.4542.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.606+0000 7f544effd640 3 client.4542.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:13.611 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.606+0000 7f544effd640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:42:13.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.606+0000 7f545dd9b640 10 monclient: get_auth_request con 0x7f544006dc40 auth_method 0 2024-03-29T11:42:13.612 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.607+0000 7f544effd640 10 client.4542.objecter ms_handle_connect 0x7f544006dc40 2024-03-29T11:42:13.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.612+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5450050680 con 0x7f54600b4200 2024-03-29T11:42:13.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.612+0000 7f544effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:13.615 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.612+0000 7f544effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:13.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.936+0000 7f544cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:13.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.936+0000 7f544cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:13.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.936+0000 7f544cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:13.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.936+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54601410a0 con 0x7f54600b4200 2024-03-29T11:42:13.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.937+0000 7f544effd640 1 -- 172.21.15.71:0/2270165313 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f54500525b0 con 0x7f54600b4200 2024-03-29T11:42:13.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.937+0000 7f544effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:13.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.937+0000 7f544effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:13.942 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:13.942 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:13.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 10 librados: watch_flush enter 2024-03-29T11:42:13.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 10 librados: watch_flush exit 2024-03-29T11:42:13.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:42:13.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:42:13.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 >> v1:172.21.15.71:6816/1596514235 conn(0x7f544006dc40 legacy=0x7f5440070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 10 monclient: shutdown 2024-03-29T11:42:13.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:13.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 >> v1:172.21.15.71:6789/0 conn(0x7f54600b4200 legacy=0x7f54600adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:13.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 shutdown_connections 2024-03-29T11:42:13.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.941+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 >> 172.21.15.71:0/2270165313 conn(0x7f546006d230 msgr2=0x7f54600b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:13.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.942+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 shutdown_connections 2024-03-29T11:42:13.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.942+0000 7f544cff9640 1 -- 172.21.15.71:0/2270165313 wait complete. 2024-03-29T11:42:13.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:13.943+0000 7f544cff9640 1 librados: shutdown 2024-03-29T11:42:13.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:20.770 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.917+0000 7fe144657640 10 monclient: get_monmap_and_config 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.917+0000 7fe144657640 10 monclient: build_initial_monmap 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.917+0000 7fe144657640 10 monclient: monmap: 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:20.918108+0000 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:20.918108+0000 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.921 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:20.922 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:20.922 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:20.922 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.918+0000 7fe144657640 1 Processor -- start 2024-03-29T11:42:20.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.919+0000 7fe144657640 1 -- start start 2024-03-29T11:42:20.922 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.919+0000 7fe144657640 10 monclient: init 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.921+0000 7fe144657640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.921+0000 7fe144657640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.921+0000 7fe144657640 10 monclient(hunting): picked mon.noname-c con 0x7fe13c0b0770 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.921+0000 7fe144657640 10 monclient(hunting): picked mon.noname-a con 0x7fe13c14a700 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe144657640 10 monclient(hunting): picked mon.noname-b con 0x7fe13c0ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe144657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c0aba70 con 0x7fe13c14a700 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe144657640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c0aecb0 con 0x7fe13c0b0770 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe144657640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c0af150 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.925 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe144657640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:20.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe144657640 10 monclient(hunting): authenticate will time out at 1395.076246s 2024-03-29T11:42:20.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe1423f3640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe13c0b0770 0x7fe13c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:37760/0 (socket says 172.21.15.71:37760) 2024-03-29T11:42:20.926 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.922+0000 7fe1423f3640 1 -- 172.21.15.71:0/795199001 learned_addr learned my addr 172.21.15.71:0/795199001 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:20.927 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695875509 0 0) 0x7fe13c0aecb0 con 0x7fe13c0b0770 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe128003520 con 0x7fe13c0b0770 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635198553 0 0) 0x7fe13c0aba70 con 0x7fe13c14a700 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13c0aecb0 con 0x7fe13c14a700 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439665279 0 0) 0x7fe13c0af150 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.928 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13c0aba70 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335005741 0 0) 0x7fe128003520 con 0x7fe13c0b0770 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 >> v1:172.21.15.71:6790/0 conn(0x7fe13c0ac5a0 legacy=0x7fe13c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 >> v1:172.21.15.71:6789/0 conn(0x7fe13c14a700 legacy=0x7fe13c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: _finish_hunting 0 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 monclient: found mon.noname-c 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0ac9d0 con 0x7fe13c0b0770 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: _finish_auth 0 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: _check_auth_tickets 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:20.929 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe134001960 con 0x7fe13c0b0770 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe1413f1640 10 monclient: dump: 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.930 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe144657640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.923+0000 7fe144657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe134001ac0 con 0x7fe13c0b0770 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe1413f1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe1413f1640 1 -- 172.21.15.71:0/795199001 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe134001ed0 con 0x7fe13c0b0770 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe1413f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe1413f1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe1413f1640 10 monclient: dump: 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:20.931 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe144657640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.924+0000 7fe144657640 4 set_mon_vals no callback set 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.925+0000 7fe144657640 10 monclient: shutdown 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.925+0000 7fe144657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:20.932 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.925+0000 7fe144657640 1 -- 172.21.15.71:0/795199001 >> v1:172.21.15.81:6789/0 conn(0x7fe13c0b0770 legacy=0x7fe13c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.925+0000 7fe144657640 1 -- 172.21.15.71:0/795199001 shutdown_connections 2024-03-29T11:42:20.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.925+0000 7fe144657640 1 -- 172.21.15.71:0/795199001 >> 172.21.15.71:0/795199001 conn(0x7fe13c06b660 msgr2=0x7fe13c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:20.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.926+0000 7fe144657640 1 -- 172.21.15.71:0/795199001 shutdown_connections 2024-03-29T11:42:20.938 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.926+0000 7fe144657640 1 -- 172.21.15.71:0/795199001 wait complete. 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.927+0000 7fe144657640 10 monclient: build_initial_monmap 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.927+0000 7fe144657640 10 monclient: monmap: 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.928+0000 7fe144657640 1 librados: starting msgr at 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.928+0000 7fe144657640 1 librados: starting objecter 2024-03-29T11:42:20.939 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.929+0000 7fe144657640 1 Processor -- start 2024-03-29T11:42:20.940 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.929+0000 7fe144657640 1 -- start start 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.929+0000 7fe144657640 1 librados: setting wanted keys 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.929+0000 7fe144657640 1 librados: calling monclient init 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.929+0000 7fe144657640 10 monclient: init 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe144657640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe144657640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe144657640 10 monclient(hunting): picked mon.noname-c con 0x7fe13c0ac5a0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe144657640 10 monclient(hunting): picked mon.noname-a con 0x7fe13c0b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe144657640 10 monclient(hunting): picked mon.noname-b con 0x7fe13c14a700 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:20.941 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe144657640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c12a190 con 0x7fe13c0b0770 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe1423f3640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7fe13c0ac5a0 0x7fe13c146f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:47070/0 (socket says 172.21.15.71:47070) 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.930+0000 7fe1423f3640 1 -- 172.21.15.71:0/1822274973 learned_addr learned my addr 172.21.15.71:0/1822274973 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe144657640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c127e10 con 0x7fe13c14a700 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe144657640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe13c128440 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 client.?.objecter ms_handle_connect 0x7fe13c0ac5a0 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe144657640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe144657640 10 monclient(hunting): authenticate will time out at 1395.085217s 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 client.?.objecter ms_handle_connect 0x7fe13c0b0770 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491461240 0 0) 0x7fe13c128440 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.942 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe120004eb0 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 client.?.objecter ms_handle_connect 0x7fe13c14a700 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.931+0000 7fe132ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008254743 0 0) 0x7fe13c127e10 con 0x7fe13c14a700 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13c128440 con 0x7fe13c14a700 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073346358 0 0) 0x7fe13c12a190 con 0x7fe13c0b0770 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:42:20.943 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe13c127e10 con 0x7fe13c0b0770 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1822165474 0 0) 0x7fe120004eb0 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe13c12a190 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe134001e90 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:20.944 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.932+0000 7fe132ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067166050 0 0) 0x7fe13c128440 con 0x7fe13c14a700 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe120004eb0 con 0x7fe13c14a700 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe1380018f0 con 0x7fe13c14a700 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.945 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3672550405 0 0) 0x7fe13c12a190 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 >> v1:172.21.15.81:6789/0 conn(0x7fe13c14a700 legacy=0x7fe13c1e6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 >> v1:172.21.15.71:6789/0 conn(0x7fe13c0b0770 legacy=0x7fe13c128960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 monclient: found mon.c 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c125ba0 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:20.946 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 10 monclient: _renew_subs 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe13c1280a0 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 10 monclient: _renew_subs 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe13c147820 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.933+0000 7fe144657640 1 librados: init done 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.934+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe134004e80 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.934+0000 7fe132ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.934+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7fe134005290 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.934+0000 7fe132ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:20.947 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.934+0000 7fe132ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.934+0000 7fe132ffd640 10 monclient: dump: 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.935+0000 7fe140bf0640 4 set_mon_vals no callback set 2024-03-29T11:42:20.948 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.935+0000 7fe130ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:20.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.936+0000 7fe130ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:20.949 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.936+0000 7fe130ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.936+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe118005080 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.937+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7fe1340053f0 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe141bf2640 10 monclient: get_auth_request con 0x7fe12006dfb0 auth_method 0 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7fe134084f00 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe132ffd640 10 client.4670.objecter ms_dispatch 0x7fe13c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe132ffd640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe132ffd640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe132ffd640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-29T11:42:20.950 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.938+0000 7fe132ffd640 10 client.4670.objecter ms_handle_connect 0x7fe12006dfb0 2024-03-29T11:42:20.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.948+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1340507e0 con 0x7fe13c0ac5a0 2024-03-29T11:42:20.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.948+0000 7fe132ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:20.951 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:20.948+0000 7fe132ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:21.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.315+0000 7fe130ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:21.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.315+0000 7fe130ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:21.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.315+0000 7fe130ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:21.318 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.315+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe118005330 con 0x7fe13c0ac5a0 2024-03-29T11:42:21.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.316+0000 7fe132ffd640 1 -- 172.21.15.71:0/1822274973 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7fe134052710 con 0x7fe13c0ac5a0 2024-03-29T11:42:21.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.316+0000 7fe132ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:21.319 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.316+0000 7fe132ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:21.319 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:21.319 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:21.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.319+0000 7fe130ff9640 10 librados: watch_flush enter 2024-03-29T11:42:21.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.319+0000 7fe130ff9640 10 librados: watch_flush exit 2024-03-29T11:42:21.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-29T11:42:21.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 10 client.4670.objecter successfully canceled tick 2024-03-29T11:42:21.322 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 >> v1:172.21.15.71:6816/1596514235 conn(0x7fe12006dfb0 legacy=0x7fe120070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 10 monclient: shutdown 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 >> v1:172.21.15.71:6790/0 conn(0x7fe13c0ac5a0 legacy=0x7fe13c146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 shutdown_connections 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.320+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 >> 172.21.15.71:0/1822274973 conn(0x7fe13c06b660 msgr2=0x7fe13c147c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.321+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 shutdown_connections 2024-03-29T11:42:21.323 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.321+0000 7fe130ff9640 1 -- 172.21.15.71:0/1822274973 wait complete. 2024-03-29T11:42:21.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:21.321+0000 7fe130ff9640 1 librados: shutdown 2024-03-29T11:42:21.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:28.146 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.310+0000 7f0dc1286640 10 monclient: get_monmap_and_config 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.310+0000 7f0dc1286640 10 monclient: build_initial_monmap 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.310+0000 7f0dc1286640 10 monclient: monmap: 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:28.311822+0000 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:28.311822+0000 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:28.314 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:28.315 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:28.315 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:28.315 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:28.315 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.312+0000 7f0dc1286640 1 Processor -- start 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.312+0000 7f0dc1286640 1 -- start start 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.312+0000 7f0dc1286640 10 monclient: init 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient(hunting): picked mon.noname-c con 0x7f0dbc0b4790 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:28.316 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient(hunting): picked mon.noname-a con 0x7f0dbc0aea90 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:28.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient(hunting): picked mon.noname-b con 0x7f0dbc0b39f0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:28.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0b0c60 con 0x7f0dbc0aea90 2024-03-29T11:42:28.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0af020 con 0x7f0dbc0b4790 2024-03-29T11:42:28.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc13fbd0 con 0x7f0dbc0b39f0 2024-03-29T11:42:28.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:28.317 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.313+0000 7f0dc1286640 10 monclient(hunting): authenticate will time out at 1402.467603s 2024-03-29T11:42:28.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.317+0000 7f0dbaffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0dbc0b4790 0x7f0dbc0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:38401/0 (socket says 172.21.15.71:38401) 2024-03-29T11:42:28.320 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.317+0000 7f0dbaffd640 1 -- 172.21.15.71:0/383520744 learned_addr learned my addr 172.21.15.71:0/383520744 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:28.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.321+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299659415 0 0) 0x7f0dbc0af020 con 0x7f0dbc0b4790 2024-03-29T11:42:28.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.321+0000 7f0db9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.321+0000 7f0db9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.321+0000 7f0db9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.321+0000 7f0db9ffb640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:42:28.324 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.321+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8003520 con 0x7f0dbc0b4790 2024-03-29T11:42:28.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927199001 0 0) 0x7f0da8003520 con 0x7f0dbc0b4790 2024-03-29T11:42:28.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 >> v1:172.21.15.71:6790/0 conn(0x7f0dbc0b39f0 legacy=0x7f0dbc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.325 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 >> v1:172.21.15.71:6789/0 conn(0x7f0dbc0aea90 legacy=0x7f0dbc0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.81:6789/0 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc13fd60 con 0x7f0dbc0b4790 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f0da4001690 con 0x7f0dbc0b4790 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0db9ffb640 10 monclient: dump: 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:28.326 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0dc1286640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:42:28.327 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.322+0000 7f0dc1286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.323+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 <== mon.1 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0da40017f0 con 0x7f0dbc0b4790 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.323+0000 7f0db9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.323+0000 7f0db9ffb640 1 -- 172.21.15.71:0/383520744 <== mon.1 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f0da4001c00 con 0x7f0dbc0b4790 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.323+0000 7f0db9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.323+0000 7f0db9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.323+0000 7f0db9ffb640 10 monclient: dump: 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.328 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:28.329 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:28.329 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:28.329 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:28.329 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:28.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.326+0000 7f0dc1286640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:28.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.326+0000 7f0dc1286640 4 set_mon_vals no callback set 2024-03-29T11:42:28.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.326+0000 7f0dc1286640 10 monclient: shutdown 2024-03-29T11:42:28.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.326+0000 7f0dc1286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:28.330 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.326+0000 7f0dc1286640 1 -- 172.21.15.71:0/383520744 >> v1:172.21.15.81:6789/0 conn(0x7f0dbc0b4790 legacy=0x7f0dbc0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.329+0000 7f0dc1286640 1 -- 172.21.15.71:0/383520744 shutdown_connections 2024-03-29T11:42:28.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.329+0000 7f0dc1286640 1 -- 172.21.15.71:0/383520744 >> 172.21.15.71:0/383520744 conn(0x7f0dbc066a00 msgr2=0x7f0dbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:28.332 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.330+0000 7f0dc1286640 1 -- 172.21.15.71:0/383520744 shutdown_connections 2024-03-29T11:42:28.333 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.330+0000 7f0dc1286640 1 -- 172.21.15.71:0/383520744 wait complete. 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.331+0000 7f0dc1286640 10 monclient: build_initial_monmap 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.331+0000 7f0dc1286640 10 monclient: monmap: 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:28.337 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:28.338 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:28.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.331+0000 7f0dc1286640 1 librados: starting msgr at 2024-03-29T11:42:28.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.331+0000 7f0dc1286640 1 librados: starting objecter 2024-03-29T11:42:28.338 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.334+0000 7f0dc1286640 1 Processor -- start 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.334+0000 7f0dc1286640 1 -- start start 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.334+0000 7f0dc1286640 1 librados: setting wanted keys 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.334+0000 7f0dc1286640 1 librados: calling monclient init 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.334+0000 7f0dc1286640 10 monclient: init 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient(hunting): picked mon.noname-a con 0x7f0dbc0b39f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:28.339 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient(hunting): picked mon.noname-c con 0x7f0dbc0b4790 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:28.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient(hunting): picked mon.noname-b con 0x7f0dbc137fe0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:28.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc137260 con 0x7f0dbc0b39f0 2024-03-29T11:42:28.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc1374b0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc0ad830 con 0x7f0dbc0b4790 2024-03-29T11:42:28.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:28.340 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.335+0000 7f0dc1286640 10 monclient(hunting): authenticate will time out at 1402.489991s 2024-03-29T11:42:28.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.339+0000 7f0dbaffd640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f0dbc0b39f0 0x7f0dbc1378d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:47461/0 (socket says 172.21.15.71:47461) 2024-03-29T11:42:28.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.339+0000 7f0dbaffd640 1 -- 172.21.15.71:0/1728653307 learned_addr learned my addr 172.21.15.71:0/1728653307 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:28.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.339+0000 7f0db37fe640 10 client.?.objecter ms_handle_connect 0x7f0dbc0b39f0 2024-03-29T11:42:28.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.339+0000 7f0db37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:28.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.339+0000 7f0db37fe640 10 client.?.objecter ms_handle_connect 0x7f0dbc137fe0 2024-03-29T11:42:28.342 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.339+0000 7f0db37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:28.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611383422 0 0) 0x7f0dbc1374b0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:42:28.344 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c004eb0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.347 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2985711973 0 0) 0x7f0d9c004eb0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d9c005300 con 0x7f0dbc137fe0 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f0db4001c30 con 0x7f0dbc137fe0 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): dump: 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:28.348 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 client.?.objecter ms_handle_connect 0x7f0dbc0b4790 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.341+0000 7f0db37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168953031 0 0) 0x7f0dbc0ad830 con 0x7f0dbc0b4790 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.349 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d9c004eb0 con 0x7f0dbc0b4790 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1607561603 0 0) 0x7f0d9c005300 con 0x7f0dbc137fe0 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 >> v1:172.21.15.71:6790/0 conn(0x7f0dbc0b4790 legacy=0x7f0dbc0aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 >> v1:172.21.15.71:6789/0 conn(0x7f0dbc0b39f0 legacy=0x7f0dbc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 1 monclient: found mon.b 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.342+0000 7f0db37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0db37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0abd00 con 0x7f0dbc137fe0 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0db37fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0db37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0db37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:28.350 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0dc1286640 5 monclient: authenticate success, global_id 4567 2024-03-29T11:42:28.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0dc1286640 10 monclient: _renew_subs 2024-03-29T11:42:28.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0dc1286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:28.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc0ac300 con 0x7f0dbc137fe0 2024-03-29T11:42:28.351 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0dc1286640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.343+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0db4003990 con 0x7f0dbc137fe0 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0db37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f0db4003ca0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0db37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0db37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0db37fe640 10 monclient: dump: 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:28.352 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:28.353 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0db97fa640 4 set_mon_vals no callback set 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0dc1286640 10 monclient: _renew_subs 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0dc1286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc0ae3d0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.344+0000 7f0dc1286640 1 librados: init done 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.346+0000 7f0dc1286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.346+0000 7f0dc1286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.346+0000 7f0dc1286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.346+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dbc0ad9c0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.348+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f0db4019c70 con 0x7f0dbc137fe0 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.349+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f0db4053a20 con 0x7f0dbc137fe0 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.349+0000 7f0db37fe640 10 client.4567.objecter ms_dispatch 0x7f0dbc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:28.354 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.349+0000 7f0db37fe640 3 client.4567.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:28.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.349+0000 7f0db37fe640 3 client.4567.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:28.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.349+0000 7f0db37fe640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-29T11:42:28.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.350+0000 7f0dbaffd640 10 monclient: get_auth_request con 0x7f0d9c06dee0 auth_method 0 2024-03-29T11:42:28.355 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.353+0000 7f0db37fe640 10 client.4567.objecter ms_handle_connect 0x7f0d9c06dee0 2024-03-29T11:42:28.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.358+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0db404fcc0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.360 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.358+0000 7f0db37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:28.361 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.358+0000 7f0db37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:28.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.720+0000 7f0dc1286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:28.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.720+0000 7f0dc1286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:28.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.720+0000 7f0dc1286640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:28.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.720+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dbc0ac020 con 0x7f0dbc137fe0 2024-03-29T11:42:28.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.721+0000 7f0db37fe640 1 -- 172.21.15.71:0/1728653307 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f0db404fcc0 con 0x7f0dbc137fe0 2024-03-29T11:42:28.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.721+0000 7f0db37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:28.724 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.721+0000 7f0db37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:28.725 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:28.725 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.725+0000 7f0dc1286640 10 librados: watch_flush enter 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 10 librados: watch_flush exit 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 10 client.4567.objecter successfully canceled tick 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 >> v1:172.21.15.71:6816/1596514235 conn(0x7f0d9c06dee0 legacy=0x7f0d9c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 10 monclient: shutdown 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 >> v1:172.21.15.81:6789/0 conn(0x7f0dbc137fe0 legacy=0x7f0dbc0ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 shutdown_connections 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.726+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 >> 172.21.15.71:0/1728653307 conn(0x7f0dbc066a00 msgr2=0x7f0dbc0b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.727+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 shutdown_connections 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.727+0000 7f0dc1286640 1 -- 172.21.15.71:0/1728653307 wait complete. 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:28.728+0000 7f0dc1286640 1 librados: shutdown 2024-03-29T11:42:28.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:35.556 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.702+0000 7f608cae4640 10 monclient: get_monmap_and_config 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.702+0000 7f608cae4640 10 monclient: build_initial_monmap 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.702+0000 7f608cae4640 10 monclient: monmap: 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:35.703126+0000 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:35.703126+0000 2024-03-29T11:42:35.705 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:35.706 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:35.706 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:35.706 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:35.706 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:35.706 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:35.706 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.703+0000 7f608cae4640 1 Processor -- start 2024-03-29T11:42:35.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.703+0000 7f608cae4640 1 -- start start 2024-03-29T11:42:35.707 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.703+0000 7f608cae4640 10 monclient: init 2024-03-29T11:42:35.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.704+0000 7f608cae4640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:35.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.704+0000 7f608cae4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:35.708 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.704+0000 7f608cae4640 10 monclient(hunting): picked mon.noname-a con 0x7f608806ee40 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:35.709 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 10 monclient(hunting): picked mon.noname-b con 0x7f60880aeda0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 10 monclient(hunting): picked mon.noname-c con 0x7f608813c470 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608806fdb0 con 0x7f608806ee40 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088071090 con 0x7f608813c470 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088071370 con 0x7f60880aeda0 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.705+0000 7f608cae4640 10 monclient(hunting): authenticate will time out at 1409.859899s 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.706+0000 7f608659c640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f608806ee40 0x7f608813bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:48160/0 (socket says 172.21.15.71:48160) 2024-03-29T11:42:35.710 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.706+0000 7f608659c640 1 -- 172.21.15.71:0/3696938394 learned_addr learned my addr 172.21.15.71:0/3696938394 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:35.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.707+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482523056 0 0) 0x7f608806fdb0 con 0x7f608806ee40 2024-03-29T11:42:35.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.707+0000 7f6085d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:42:35.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.707+0000 7f6085d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:35.711 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.707+0000 7f6085d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:35.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.707+0000 7f6085d9b640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:42:35.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.707+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6074003520 con 0x7f608806ee40 2024-03-29T11:42:35.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.708+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 536895593 0 0) 0x7f6074003520 con 0x7f608806ee40 2024-03-29T11:42:35.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.708+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 >> v1:172.21.15.71:6790/0 conn(0x7f60880aeda0 legacy=0x7f6088071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.712 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.708+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 >> v1:172.21.15.81:6789/0 conn(0x7f608813c470 legacy=0x7f608814a640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:35.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:42:35.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 1 monclient: found mon.noname-a 2024-03-29T11:42:35.713 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.71:6789/0 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088070220 con 0x7f608806ee40 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: _finish_auth 0 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6070001690 con 0x7f608806ee40 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f6085d9b640 10 monclient: dump: 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.714 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f608cae4640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.710+0000 7f608cae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:35.715 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.711+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 <== mon.0 v1:172.21.15.71:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6070001860 con 0x7f608806ee40 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.711+0000 7f6085d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.711+0000 7f6085d9b640 1 -- 172.21.15.71:0/3696938394 <== mon.0 v1:172.21.15.71:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6070001cb0 con 0x7f608806ee40 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.711+0000 7f6085d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.711+0000 7f6085d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.711+0000 7f6085d9b640 10 monclient: dump: 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:35.716 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:35.717 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:35.717 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:35.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.713+0000 7f608cae4640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:35.717 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.713+0000 7f608cae4640 4 set_mon_vals no callback set 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.714+0000 7f608cae4640 10 monclient: shutdown 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.714+0000 7f608cae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.714+0000 7f608cae4640 1 -- 172.21.15.71:0/3696938394 >> v1:172.21.15.71:6789/0 conn(0x7f608806ee40 legacy=0x7f608813bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.714+0000 7f608cae4640 1 -- 172.21.15.71:0/3696938394 shutdown_connections 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.714+0000 7f608cae4640 1 -- 172.21.15.71:0/3696938394 >> 172.21.15.71:0/3696938394 conn(0x7f6088067470 msgr2=0x7f60880b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.714+0000 7f608cae4640 1 -- 172.21.15.71:0/3696938394 shutdown_connections 2024-03-29T11:42:35.718 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.715+0000 7f608cae4640 1 -- 172.21.15.71:0/3696938394 wait complete. 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.716+0000 7f608cae4640 10 monclient: build_initial_monmap 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.716+0000 7f608cae4640 10 monclient: monmap: 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:35.719 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:35.720 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:35.720 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:35.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 1 librados: starting msgr at 2024-03-29T11:42:35.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 1 librados: starting objecter 2024-03-29T11:42:35.720 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 1 Processor -- start 2024-03-29T11:42:35.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 1 -- start start 2024-03-29T11:42:35.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 1 librados: setting wanted keys 2024-03-29T11:42:35.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 1 librados: calling monclient init 2024-03-29T11:42:35.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.717+0000 7f608cae4640 10 monclient: init 2024-03-29T11:42:35.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:35.722 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient(hunting): picked mon.noname-b con 0x7f608806ee40 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient(hunting): picked mon.noname-c con 0x7f60880aeda0 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient(hunting): picked mon.noname-a con 0x7f608813c470 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608813f7d0 con 0x7f608813c470 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60881d70c0 con 0x7f608806ee40 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60881410d0 con 0x7f60880aeda0 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:35.723 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.718+0000 7f608cae4640 10 monclient(hunting): authenticate will time out at 1409.872597s 2024-03-29T11:42:35.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.722+0000 7f6086d9d640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f608813c470 0x7f60881df460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:49140/0 (socket says 172.21.15.71:49140) 2024-03-29T11:42:35.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.722+0000 7f6086d9d640 1 -- 172.21.15.71:0/143328928 learned_addr learned my addr 172.21.15.71:0/143328928 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:35.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.722+0000 7f607f7fe640 10 client.?.objecter ms_handle_connect 0x7f608813c470 2024-03-29T11:42:35.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.722+0000 7f607f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:35.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.722+0000 7f607f7fe640 10 client.?.objecter ms_handle_connect 0x7f608806ee40 2024-03-29T11:42:35.729 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.722+0000 7f607f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445037676 0 0) 0x7f608813f7d0 con 0x7f608813c470 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6068004eb0 con 0x7f608813c470 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1693198947 0 0) 0x7f6068004eb0 con 0x7f608813c470 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f608813f7d0 con 0x7f608813c470 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f6080001750 con 0x7f608813c470 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:35.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): dump: 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:35.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 153372858 0 0) 0x7f608813f7d0 con 0x7f608813c470 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 >> v1:172.21.15.71:6790/0 conn(0x7f60880aeda0 legacy=0x7f6088133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 >> v1:172.21.15.81:6789/0 conn(0x7f608806ee40 legacy=0x7f60881d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 monclient: found mon.a 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608813cb20 con 0x7f608813c470 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f607f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 10 monclient: _renew_subs 2024-03-29T11:42:35.732 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f608813cd10 con 0x7f608813c470 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 10 monclient: _renew_subs 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60881d7eb0 con 0x7f608813c470 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.723+0000 7f608cae4640 1 librados: init done 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.724+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6080001ce0 con 0x7f608813c470 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.724+0000 7f607f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.724+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f60800049d0 con 0x7f608813c470 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.724+0000 7f607f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.724+0000 7f607f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.724+0000 7f607f7fe640 10 monclient: dump: 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:35.733 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:35.734 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.726+0000 7f608559a640 4 set_mon_vals no callback set 2024-03-29T11:42:35.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f6080004b30 con 0x7f608813c470 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f6080084980 con 0x7f608813c470 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f607f7fe640 10 client.4560.objecter ms_dispatch 0x7f6088000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f607f7fe640 3 client.4560.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f607f7fe640 3 client.4560.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f607f7fe640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.727+0000 7f608659c640 10 monclient: get_auth_request con 0x7f606806dbd0 auth_method 0 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.728+0000 7f608cae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.728+0000 7f608cae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.728+0000 7f608cae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.728+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c005080 con 0x7f608813c470 2024-03-29T11:42:35.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.728+0000 7f607f7fe640 10 client.4560.objecter ms_handle_connect 0x7f606806dbd0 2024-03-29T11:42:35.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.736+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6080050260 con 0x7f608813c470 2024-03-29T11:42:35.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.736+0000 7f607f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:35.739 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:35.736+0000 7f607f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:36.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.079+0000 7f608cae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:36.082 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.079+0000 7f608cae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:36.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.079+0000 7f608cae4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.71:6789/0 2024-03-29T11:42:36.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.079+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 --> v1:172.21.15.71:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f606c005330 con 0x7f608813c470 2024-03-29T11:42:36.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.082+0000 7f607f7fe640 1 -- 172.21.15.71:0/143328928 <== mon.0 v1:172.21.15.71:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f6080052190 con 0x7f608813c470 2024-03-29T11:42:36.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.082+0000 7f607f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:36.085 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.082+0000 7f607f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:36.085 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:36.085 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.088+0000 7f608cae4640 10 librados: watch_flush enter 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.088+0000 7f608cae4640 10 librados: watch_flush exit 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.088+0000 7f608cae4640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 >> v1:172.21.15.71:6816/1596514235 conn(0x7f606806dbd0 legacy=0x7f6068070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 10 monclient: shutdown 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 >> v1:172.21.15.71:6789/0 conn(0x7f608813c470 legacy=0x7f60881df460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 shutdown_connections 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 >> 172.21.15.71:0/143328928 conn(0x7f6088067470 msgr2=0x7f608813ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:36.092 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 shutdown_connections 2024-03-29T11:42:36.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 -- 172.21.15.71:0/143328928 wait complete. 2024-03-29T11:42:36.093 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:36.089+0000 7f608cae4640 1 librados: shutdown 2024-03-29T11:42:36.111 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:42.919 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.054+0000 7f429a582640 10 monclient: get_monmap_and_config 2024-03-29T11:42:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.054+0000 7f429a582640 10 monclient: build_initial_monmap 2024-03-29T11:42:43.058 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.054+0000 7f429a582640 10 monclient: monmap: 2024-03-29T11:42:43.058 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:43.058 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:43.058 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:43.056281+0000 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:43.056281+0000 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:43.059 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.055+0000 7f429a582640 1 Processor -- start 2024-03-29T11:42:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.055+0000 7f429a582640 1 -- start start 2024-03-29T11:42:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.055+0000 7f429a582640 10 monclient: init 2024-03-29T11:42:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:43.060 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient(hunting): picked mon.noname-c con 0x7f42941457f0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient(hunting): picked mon.noname-a con 0x7f429413f650 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient(hunting): picked mon.noname-b con 0x7f42940a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42940aaef0 con 0x7f429413f650 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4294145bd0 con 0x7f42941457f0 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4294145eb0 con 0x7f42940a9d70 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f429a582640 10 monclient(hunting): authenticate will time out at 1417.211562s 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f42937fe640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f429413f650 0x7f42940b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:35384/0 (socket says 172.21.15.71:35384) 2024-03-29T11:42:43.061 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.056+0000 7f42937fe640 1 -- 172.21.15.71:0/1880870238 learned_addr learned my addr 172.21.15.71:0/1880870238 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246482621 0 0) 0x7f4294145eb0 con 0x7f42940a9d70 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f427c003520 con 0x7f42940a9d70 2024-03-29T11:42:43.062 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472739814 0 0) 0x7f427c003520 con 0x7f42940a9d70 2024-03-29T11:42:43.063 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.058+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 >> v1:172.21.15.81:6789/0 conn(0x7f42941457f0 legacy=0x7f429413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:43.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 >> v1:172.21.15.71:6789/0 conn(0x7f429413f650 legacy=0x7f42940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:43.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 1 monclient: found mon.noname-b 2024-03-29T11:42:43.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:43.065 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294146340 con 0x7f42940a9d70 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f428c001a10 con 0x7f42940a9d70 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.060+0000 7f4292ffd640 10 monclient: dump: 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.066 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f428c001be0 con 0x7f42940a9d70 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f4292ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f4292ffd640 1 -- 172.21.15.71:0/1880870238 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f428c001ff0 con 0x7f42940a9d70 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f4292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f4292ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f4292ffd640 10 monclient: dump: 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.067 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f429a582640 5 monclient: authenticate success, global_id 4676 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f429a582640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:43.068 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.061+0000 7f429a582640 4 set_mon_vals no callback set 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.062+0000 7f429a582640 10 monclient: shutdown 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.062+0000 7f429a582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.062+0000 7f429a582640 1 -- 172.21.15.71:0/1880870238 >> v1:172.21.15.71:6790/0 conn(0x7f42940a9d70 legacy=0x7f429414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.062+0000 7f429a582640 1 -- 172.21.15.71:0/1880870238 shutdown_connections 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.062+0000 7f429a582640 1 -- 172.21.15.71:0/1880870238 >> 172.21.15.71:0/1880870238 conn(0x7f4294066a00 msgr2=0x7f4294066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.063+0000 7f429a582640 1 -- 172.21.15.71:0/1880870238 shutdown_connections 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.063+0000 7f429a582640 1 -- 172.21.15.71:0/1880870238 wait complete. 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.064+0000 7f429a582640 10 monclient: build_initial_monmap 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.064+0000 7f429a582640 10 monclient: monmap: 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:43.070 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 1 librados: starting msgr at 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 1 librados: starting objecter 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 1 Processor -- start 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 1 -- start start 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 1 librados: setting wanted keys 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 1 librados: calling monclient init 2024-03-29T11:42:43.071 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.065+0000 7f429a582640 10 monclient: init 2024-03-29T11:42:43.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:43.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:43.072 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient(hunting): picked mon.noname-c con 0x7f42940a9d70 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient(hunting): picked mon.noname-b con 0x7f429413f650 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient(hunting): picked mon.noname-a con 0x7f42941457f0 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429413b0e0 con 0x7f42941457f0 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4294071150 con 0x7f429413f650 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4294070380 con 0x7f42940a9d70 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f4293fff640 1 --1- >> v1:172.21.15.71:6790/0 conn(0x7f42940a9d70 0x7f4294070a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.71:6790/0 says I am v1:172.21.15.71:34272/0 (socket says 172.21.15.71:34272) 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f4293fff640 1 -- 172.21.15.71:0/3462955512 learned_addr learned my addr 172.21.15.71:0/3462955512 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f4290ff9640 10 client.?.objecter ms_handle_connect 0x7f42940a9d70 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f4290ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:43.074 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.066+0000 7f429a582640 10 monclient(hunting): authenticate will time out at 1417.221550s 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654197938 0 0) 0x7f4294070380 con 0x7f42940a9d70 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4280004eb0 con 0x7f42940a9d70 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4175245266 0 0) 0x7f4280004eb0 con 0x7f42940a9d70 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4294070380 con 0x7f42940a9d70 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f42840017d0 con 0x7f42940a9d70 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:43.075 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 204820427 0 0) 0x7f4294070380 con 0x7f42940a9d70 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.067+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 >> v1:172.21.15.81:6789/0 conn(0x7f429413f650 legacy=0x7f429413a2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 >> v1:172.21.15.71:6789/0 conn(0x7f42941457f0 legacy=0x7f429413a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 1 monclient: found mon.c 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:43.076 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429406f100 con 0x7f42940a9d70 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 client.?.objecter ms_handle_connect 0x7f42941457f0 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 10 monclient: _renew_subs 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4294142790 con 0x7f42940a9d70 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 10 monclient: _renew_subs 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42941ded00 con 0x7f42940a9d70 2024-03-29T11:42:43.077 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f429a582640 1 librados: init done 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4284001930 con 0x7f42940a9d70 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f4284004a40 con 0x7f42940a9d70 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.068+0000 7f4290ff9640 10 monclient: dump: 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:43.078 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:43.079 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:43.079 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.069+0000 7f42927fc640 4 set_mon_vals no callback set 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.071+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f4284004c20 con 0x7f42940a9d70 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.071+0000 7f429a582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.071+0000 7f429a582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.071+0000 7f429a582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.071+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4274005080 con 0x7f42940a9d70 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f4284084b00 con 0x7f42940a9d70 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f4290ff9640 10 client.4679.objecter ms_dispatch 0x7f4294000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f4290ff9640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:43.080 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f4290ff9640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f4290ff9640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T11:42:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f42937fe640 10 monclient: get_auth_request con 0x7f428006db10 auth_method 0 2024-03-29T11:42:43.081 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.072+0000 7f4290ff9640 10 client.4679.objecter ms_handle_connect 0x7f428006db10 2024-03-29T11:42:43.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.080+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4284050210 con 0x7f42940a9d70 2024-03-29T11:42:43.083 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.080+0000 7f4290ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:43.084 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.080+0000 7f4290ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:43.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.404+0000 7f429a582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:43.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.404+0000 7f429a582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:43.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.404+0000 7f429a582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.71:6790/0 2024-03-29T11:42:43.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.404+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 --> v1:172.21.15.71:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4274005330 con 0x7f42940a9d70 2024-03-29T11:42:43.411 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:43.411 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:43.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.408+0000 7f4290ff9640 1 -- 172.21.15.71:0/3462955512 <== mon.2 v1:172.21.15.71:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f4284052310 con 0x7f42940a9d70 2024-03-29T11:42:43.411 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.408+0000 7f4290ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:43.412 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.408+0000 7f4290ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:43.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.411+0000 7f429a582640 10 librados: watch_flush enter 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.411+0000 7f429a582640 10 librados: watch_flush exit 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 10 client.4679.objecter successfully canceled tick 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 >> v1:172.21.15.71:6816/1596514235 conn(0x7f428006db10 legacy=0x7f428006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 10 monclient: shutdown 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 >> v1:172.21.15.71:6790/0 conn(0x7f42940a9d70 legacy=0x7f4294070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 shutdown_connections 2024-03-29T11:42:43.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 >> 172.21.15.71:0/3462955512 conn(0x7f4294066a00 msgr2=0x7f42940b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:43.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 shutdown_connections 2024-03-29T11:42:43.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.412+0000 7f429a582640 1 -- 172.21.15.71:0/3462955512 wait complete. 2024-03-29T11:42:43.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:43.413+0000 7f429a582640 1 librados: shutdown 2024-03-29T11:42:43.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:50.239 DEBUG:teuthology.orchestra.run.smithi071:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:50.387 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.384+0000 7f987bd9c640 10 monclient: get_monmap_and_config 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.384+0000 7f987bd9c640 10 monclient: build_initial_monmap 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.384+0000 7f987bd9c640 10 monclient: monmap: 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:42:50.385655+0000 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:42:50.385655+0000 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-c 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-b 2024-03-29T11:42:50.388 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.389 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.386+0000 7f987bd9c640 1 Processor -- start 2024-03-29T11:42:50.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.387+0000 7f987bd9c640 1 -- start start 2024-03-29T11:42:50.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.387+0000 7f987bd9c640 10 monclient: init 2024-03-29T11:42:50.390 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient(hunting): picked mon.noname-a con 0x7f98740b0770 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient(hunting): picked mon.noname-b con 0x7f987414a700 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient(hunting): picked mon.noname-c con 0x7f98740ac5a0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 1 -- --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98740aba70 con 0x7f98740b0770 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98740aecb0 con 0x7f98740ac5a0 2024-03-29T11:42:50.391 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 1 -- --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98740af150 con 0x7f987414a700 2024-03-29T11:42:50.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:50.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f987bd9c640 10 monclient(hunting): authenticate will time out at 1424.542266s 2024-03-29T11:42:50.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f9879b38640 1 --1- >> v1:172.21.15.71:6789/0 conn(0x7f98740b0770 0x7f9874149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.71:6789/0 says I am v1:172.21.15.71:37844/0 (socket says 172.21.15.71:37844) 2024-03-29T11:42:50.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.388+0000 7f9879b38640 1 -- 172.21.15.71:0/1775013851 learned_addr learned my addr 172.21.15.71:0/1775013851 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:50.392 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894136041 0 0) 0x7f98740aba70 con 0x7f98740b0770 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9860003520 con 0x7f98740b0770 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4856601 0 0) 0x7f98740af150 con 0x7f987414a700 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98740aba70 con 0x7f987414a700 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3251197710 0 0) 0x7f98740aba70 con 0x7f987414a700 2024-03-29T11:42:50.393 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.389+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 >> v1:172.21.15.81:6789/0 conn(0x7f98740ac5a0 legacy=0x7f9874151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:50.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 >> v1:172.21.15.71:6789/0 conn(0x7f98740b0770 legacy=0x7f9874149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: _finish_hunting 0 2024-03-29T11:42:50.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 1 monclient: found mon.noname-b 2024-03-29T11:42:50.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:50.394 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.71:6790/0 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 --> v1:172.21.15.71:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98740ac9d0 con 0x7f987414a700 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: _finish_auth 0 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: _check_auth_tickets 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f986c0019f0 con 0x7f987414a700 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f9878b36640 10 monclient: dump: 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.395 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.396 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:50.396 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:50.396 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:50.396 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f987bd9c640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:42:50.396 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.391+0000 7f987bd9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 <== mon.2 v1:172.21.15.71:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f986c001b50 con 0x7f987414a700 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f9878b36640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f9878b36640 1 -- 172.21.15.71:0/1775013851 <== mon.2 v1:172.21.15.71:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f986c001fa0 con 0x7f987414a700 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f9878b36640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f9878b36640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f9878b36640 10 monclient: dump: 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:50.397 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f987bd9c640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.392+0000 7f987bd9c640 4 set_mon_vals no callback set 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.393+0000 7f987bd9c640 10 monclient: shutdown 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.393+0000 7f987bd9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.393+0000 7f987bd9c640 1 -- 172.21.15.71:0/1775013851 >> v1:172.21.15.71:6790/0 conn(0x7f987414a700 legacy=0x7f98740abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.393+0000 7f987bd9c640 1 -- 172.21.15.71:0/1775013851 shutdown_connections 2024-03-29T11:42:50.398 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.393+0000 7f987bd9c640 1 -- 172.21.15.71:0/1775013851 >> 172.21.15.71:0/1775013851 conn(0x7f987406b660 msgr2=0x7f987406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:50.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.394+0000 7f987bd9c640 1 -- 172.21.15.71:0/1775013851 shutdown_connections 2024-03-29T11:42:50.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.394+0000 7f987bd9c640 1 -- 172.21.15.71:0/1775013851 wait complete. 2024-03-29T11:42:50.399 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 10 monclient: build_initial_monmap 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 10 monclient: monmap: 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:epoch 0 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:last_changed 0.000000 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:created 0.000000 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.noname-a 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.noname-b 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.noname-c 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 1 librados: starting msgr at 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 1 librados: starting objecter 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 1 Processor -- start 2024-03-29T11:42:50.400 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 1 -- start start 2024-03-29T11:42:50.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 1 librados: setting wanted keys 2024-03-29T11:42:50.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 1 librados: calling monclient init 2024-03-29T11:42:50.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.395+0000 7f987bd9c640 10 monclient: init 2024-03-29T11:42:50.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.396+0000 7f987bd9c640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:50.401 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.396+0000 7f987bd9c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:50.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.398+0000 7f987bd9c640 10 monclient(hunting): picked mon.noname-b con 0x7f98740ac5a0 addr v1:172.21.15.81:6789/0 2024-03-29T11:42:50.402 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f9879b38640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f98740ac5a0 0x7f98741e47f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.71:33665/0 (socket says 172.21.15.71:33665) 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 10 monclient(hunting): picked mon.noname-c con 0x7f98740b0770 addr v1:172.21.15.71:6790/0 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f9879b38640 1 -- 172.21.15.71:0/540886215 learned_addr learned my addr 172.21.15.71:0/540886215 (peer_addr_for_me v1:172.21.15.71:0/0) 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 10 monclient(hunting): picked mon.noname-a con 0x7f987414a700 addr v1:172.21.15.71:6789/0 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.71:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98741e4080 con 0x7f987414a700 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98741e4f00 con 0x7f98740ac5a0 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f986a7fc640 10 client.?.objecter ms_handle_connect 0x7f98740b0770 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f986a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f986a7fc640 10 client.?.objecter ms_handle_connect 0x7f98740ac5a0 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f986a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.71:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98741e42d0 con 0x7f98740b0770 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.399+0000 7f987bd9c640 10 monclient(hunting): authenticate will time out at 1424.553745s 2024-03-29T11:42:50.403 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 client.?.objecter ms_handle_connect 0x7f987414a700 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048436777 0 0) 0x7f98741e4f00 con 0x7f98740ac5a0 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858004eb0 con 0x7f98740ac5a0 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.2 v1:172.21.15.71:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493331611 0 0) 0x7f98741e42d0 con 0x7f98740b0770 2024-03-29T11:42:50.405 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.71:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98741e4f00 con 0x7f98740b0770 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.0 v1:172.21.15.71:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942846598 0 0) 0x7f98741e4080 con 0x7f987414a700 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.71:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98741e42d0 con 0x7f987414a700 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 899531250 0 0) 0x7f9858004eb0 con 0x7f98740ac5a0 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98741e4080 con 0x7f98740ac5a0 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9870001a70 con 0x7f98740ac5a0 2024-03-29T11:42:50.406 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:50.407 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.2 v1:172.21.15.71:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1624401538 0 0) 0x7f98741e4f00 con 0x7f98740b0770 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.71:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9858004eb0 con 0x7f98740b0770 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.2 v1:172.21.15.71:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f986c001e60 con 0x7f98740b0770 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.400+0000 7f986a7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:50.408 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2264649042 0 0) 0x7f98741e4080 con 0x7f98740ac5a0 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 >> v1:172.21.15.71:6790/0 conn(0x7f98740b0770 legacy=0x7f987411fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 >> v1:172.21.15.71:6789/0 conn(0x7f987414a700 legacy=0x7f9874147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 monclient: found mon.b 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9874120170 con 0x7f98740ac5a0 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 10 monclient: _renew_subs 2024-03-29T11:42:50.409 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f987411eff0 con 0x7f98740ac5a0 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 10 monclient: _renew_subs 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98700047e0 con 0x7f98740ac5a0 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9874148670 con 0x7f98740ac5a0 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 866418482 0 0) 0x7f9870004af0 con 0x7f98740ac5a0 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986a7fc640 10 monclient: dump: 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:epoch 1 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:fsid 4e4d801d-5e35-484d-91ad-f481aed4a661 2024-03-29T11:42:50.410 INFO:teuthology.misc.health.smithi071.stderr:last_changed 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.411 INFO:teuthology.misc.health.smithi071.stderr:created 2024-03-29T11:31:15.067464+0000 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:election_strategy: 1 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:0: v1:172.21.15.71:6789/0 mon.a 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:1: v1:172.21.15.81:6789/0 mon.b 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:2: v1:172.21.15.71:6790/0 mon.c 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr: 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f987bd9c640 1 librados: init done 2024-03-29T11:42:50.414 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.401+0000 7f986b7fe640 4 set_mon_vals no callback set 2024-03-29T11:42:50.415 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.403+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2569682461 0 0) 0x7f987001aac0 con 0x7f98740ac5a0 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.403+0000 7f987bd9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.403+0000 7f987bd9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.403+0000 7f987bd9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.403+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f985c005080 con 0x7f98740ac5a0 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.404+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523027014 0 0) 0x7f9870084b30 con 0x7f98740ac5a0 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.404+0000 7f986a7fc640 10 client.4570.objecter ms_dispatch 0x7f9874000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.404+0000 7f986a7fc640 3 client.4570.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.404+0000 7f986a7fc640 3 client.4570.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.404+0000 7f986a7fc640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.404+0000 7f9879337640 10 monclient: get_auth_request con 0x7f985806df50 auth_method 0 2024-03-29T11:42:50.416 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.405+0000 7f986a7fc640 10 client.4570.objecter ms_handle_connect 0x7f985806df50 2024-03-29T11:42:50.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.411+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9870050240 con 0x7f98740ac5a0 2024-03-29T11:42:50.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.411+0000 7f986a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:50.417 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.411+0000 7f986a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.726+0000 7f987bd9c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.727+0000 7f987bd9c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.727+0000 7f987bd9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.81:6789/0 2024-03-29T11:42:50.730 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.727+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f985c005330 con 0x7f98740ac5a0 2024-03-29T11:42:50.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.728+0000 7f986a7fc640 1 -- 172.21.15.71:0/540886215 <== mon.1 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3513915834) 0x7f9870052340 con 0x7f98740ac5a0 2024-03-29T11:42:50.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.728+0000 7f986a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:50.731 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.728+0000 7f986a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:50.731 INFO:teuthology.misc.health.smithi071.stdout: 2024-03-29T11:42:50.731 INFO:teuthology.misc.health.smithi071.stdout:{"epoch":11,"fsid":"4e4d801d-5e35-484d-91ad-f481aed4a661","created":"2024-03-29T11:31:53.216881+0000","modified":"2024-03-29T11:41:58.143195+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:41:58.143195+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"494f788b-24f0-4777-9c63-fe989d2c07ba","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e16a4a37-aefb-458a-b2d9-367c94929241","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7f9ac3d9-b877-4b68-9183-b33976111dd1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"4cfb3619-8b50-4c13-a7cb-d83b30da8e7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"cb4e64eb-da0c-4f86-a86a-af9c7cf0c55a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ac76de25-e09f-44d1-841c-a04fa0b0590f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"26f4dbac-a9d5-44a1-8da6-6c1c0ce4f425","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"eff27a86-368e-495c-8e23-1a1bcd717839","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:50.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 10 librados: watch_flush enter 2024-03-29T11:42:50.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 10 librados: watch_flush exit 2024-03-29T11:42:50.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-29T11:42:50.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 10 client.4570.objecter successfully canceled tick 2024-03-29T11:42:50.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 >> v1:172.21.15.71:6816/1596514235 conn(0x7f985806df50 legacy=0x7f9858070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.735 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 10 monclient: shutdown 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 >> v1:172.21.15.81:6789/0 conn(0x7f98740ac5a0 legacy=0x7f98741e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 shutdown_connections 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.731+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 >> 172.21.15.71:0/540886215 conn(0x7f987406b660 msgr2=0x7f9874071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.732+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 shutdown_connections 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.732+0000 7f987bd9c640 1 -- 172.21.15.71:0/540886215 wait complete. 2024-03-29T11:42:50.736 INFO:teuthology.misc.health.smithi071.stderr:2024-03-29T11:42:50.732+0000 7f987bd9c640 1 librados: shutdown 2024-03-29T11:42:50.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:50.752 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:42:50.754 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:42:50.754 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:42:50.754 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:50.787 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:42:50.787 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:42:50.788 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:50.858 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:42:50.858 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:42:50.858 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:50.929 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:42:50.929 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:42:50.929 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:50.996 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:42:50.997 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:42:50.997 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.039 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:42:51.039 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:42:51.039 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.074 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:42:51.074 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:42:51.074 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.142 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:42:51.142 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:42:51.142 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.213 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:42:51.213 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:42:51.213 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:42:51.213 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.237 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:42:51.237 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:42:51.238 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.277 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:42:51.277 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:42:51.277 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:42:51.277 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.289 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:42:51.289 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:42:51.289 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.341 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:42:51.341 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:42:51.341 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:42:51.352 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:42:51.352 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:42:51.352 DEBUG:teuthology.orchestra.run.smithi071:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:42:51.412 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:42:51.413 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:42:52.216 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:42:52.216 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:42:52.342 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:42:52.343 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:42:52.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:42:52.401 DEBUG:teuthology.orchestra.run.smithi071:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:42:52.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:42:52.474 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:42:54.976 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:42:54.977 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:42:55.086 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:42:55.087 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:42:55.145 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:42:55.145 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:42:55.217 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:42:55.219 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/data/mon.a.tgz 2024-03-29T11:42:55.219 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-29T11:42:55.238 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.BoUQL95eYZ 2024-03-29T11:42:55.238 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BoUQL95eYZ 2024-03-29T11:42:55.359 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.BoUQL95eYZ 2024-03-29T11:42:55.445 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.BoUQL95eYZ is 212KB 2024-03-29T11:42:55.539 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.BoUQL95eYZ 2024-03-29T11:42:55.555 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/data/mon.c.tgz 2024-03-29T11:42:55.555 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-29T11:42:55.609 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.c3b2VQze04 2024-03-29T11:42:55.609 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.c3b2VQze04 2024-03-29T11:42:55.709 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.c3b2VQze04 2024-03-29T11:42:55.794 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.c3b2VQze04 is 212KB 2024-03-29T11:42:55.888 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.c3b2VQze04 2024-03-29T11:42:55.902 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/data/mon.b.tgz 2024-03-29T11:42:55.902 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-29T11:42:55.920 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.knKwpXYQgm 2024-03-29T11:42:55.920 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.knKwpXYQgm 2024-03-29T11:42:56.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.knKwpXYQgm 2024-03-29T11:42:56.122 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.knKwpXYQgm is 212KB 2024-03-29T11:42:56.188 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.knKwpXYQgm 2024-03-29T11:42:56.203 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:42:56.203 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:42:56.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:42:56.286 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:42:56.324 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:42:56.324 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:42:56.326 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/remote/smithi071/crash 2024-03-29T11:42:56.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:42:56.356 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/remote/smithi081/crash 2024-03-29T11:42:56.357 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:42:56.390 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:42:56.390 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:42:56.397 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:42:56.421 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38533.log 2024-03-29T11:42:56.421 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-03-29T11:42:56.421 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34814.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34814.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34814.log.gz 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34694.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34574.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:42:56.474 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34734.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34694.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34734.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34694.log.gz 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34614.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34734.log.gz 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34774.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34854.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34614.log.gz 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34774.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.34854.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34774.log.gz 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34854.log.gz 2024-03-29T11:42:56.475 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:42:56.476 INFO:teuthology.orchestra.run.smithi081.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38533.log.gz 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39976.log 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gz 2024-03-29T11:42:56.480 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38806.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39040.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39976.log.gz 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40916.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40838.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38806.log.gz 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38219.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39040.log.gz 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40838.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40916.log.gz 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39703.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40838.log.gz 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41073.log 2024-03-29T11:42:56.481 INFO:teuthology.orchestra.run.smithi071.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38219.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39703.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39820.log 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39469.log: gzip 85.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.41073.log.gz -- 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr: /var/log/ceph/ceph-client.admin.39391.log 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39820.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34922.log 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39820.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.34922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40015.log 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40523.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34922.log.gz 2024-03-29T11:42:56.482 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40015.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41112.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40015.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.audit.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40601.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38494.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41112.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39118.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41580.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gz 2024-03-29T11:42:56.483 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38728.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38494.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39859.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39118.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34819.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38728.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38611.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39898.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39079.log 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34819.log.gz 2024-03-29T11:42:56.484 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40093.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39898.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39079.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39274.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39235.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40093.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39352.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39274.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40210.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39352.log.gz 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38845.log 2024-03-29T11:42:56.485 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40799.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39430.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38258.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40799.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38845.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39508.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38336.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38258.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40799.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41385.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38258.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38336.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39508.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40682.log 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41385.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38336.log.gz 2024-03-29T11:42:56.486 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41385.log.gz 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40877.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40682.log.gz 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39001.log 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.38962.log: -- /var/log/ceph/ceph-client.admin.41151.log 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39625.log 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-03-29T11:42:56.487 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41151.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.39001.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41151.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39547.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41190.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39625.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41463.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34528.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39547.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.34528.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41463.log.gz 2024-03-29T11:42:56.488 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40484.log 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34528.log.gz 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41619.log 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40249.log 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41619.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40484.log.gz 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph.tmp-client.admin.28709.log 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr: 85.7%/var/log/ceph/ceph-client.admin.40249.log: -- replaced with /var/log/ceph/ceph-client.admin.41619.log.gz 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40760.log 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph.tmp-client.admin.28709.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28709.log.gz 2024-03-29T11:42:56.489 INFO:teuthology.orchestra.run.smithi071.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40249.log.gz 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41541.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40760.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41541.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40760.log.gz 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41424.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41541.log.gz 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40721.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38297.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40721.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41424.log.gz 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38572.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:gzip 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40721.log.gz 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38767.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38297.log: /var/log/ceph/ceph-client.admin.38572.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:42:56.490 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38767.log: 87.1% 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38297.log.gz 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38572.log.gz 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38767.log.gz 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41268.log/var/log/ceph/ceph.log: 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41307.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41268.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr: -- /var/log/ceph/ceph-client.admin.38455.log 2024-03-29T11:42:56.491 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-osd.2.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41268.log.gz 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40995.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38455.log.gz 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39781.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38414.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40640.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39781.log.gz 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-03-29T11:42:56.492 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38414.log: /var/log/ceph/ceph-client.admin.39157.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38414.log.gz 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38884.log 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.41502.log: /var/log/ceph/ceph-client.admin.38884.log: 86.2% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38884.log.gz 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38650.log 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39586.log: /var/log/ceph/ceph-client.admin.39937.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-03-29T11:42:56.493 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.38650.log: /var/log/ceph/ceph-client.admin.41346.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38650.log.gz 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38689.log 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41229.log 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40132.log: /var/log/ceph/ceph-client.admin.38689.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38689.log.gz 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40327.log 2024-03-29T11:42:56.494 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39196.log 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40327.log: /var/log/ceph/ceph-client.admin.41229.log: 86.1% 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41229.log.gz 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40327.log.gz 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40366.log 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40288.log 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.39196.log: /var/log/ceph/ceph-client.admin.40366.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39196.log.gz 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40366.log.gz 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40445.log 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40445.log: /var/log/ceph/ceph-client.admin.40288.log: 86.5% 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40288.log.gz 2024-03-29T11:42:56.495 INFO:teuthology.orchestra.run.smithi071.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40445.log.gz 2024-03-29T11:42:56.496 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38923.log 2024-03-29T11:42:56.496 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:42:56.496 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-client.admin.40054.log: /var/log/ceph/ceph-client.admin.38923.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2024-03-29T11:42:56.496 INFO:teuthology.orchestra.run.smithi071.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38923.log.gz 2024-03-29T11:42:56.512 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:43:08.098 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:43:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:43:08.503 INFO:teuthology.orchestra.run.smithi081.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:43:08.571 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:43:08.634 INFO:teuthology.orchestra.run.smithi081.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:43:12.368 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:43:12.482 INFO:teuthology.orchestra.run.smithi071.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:43:12.639 INFO:teuthology.orchestra.run.smithi081.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:43:45.367 INFO:teuthology.orchestra.run.smithi071.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:43:52.327 INFO:teuthology.orchestra.run.smithi071.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:43:52.329 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-29T11:43:52.329 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m55.919s 2024-03-29T11:43:52.329 INFO:teuthology.orchestra.run.smithi071.stderr:user 2m34.635s 2024-03-29T11:43:52.329 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m5.605s 2024-03-29T11:44:05.878 INFO:teuthology.orchestra.run.smithi081.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:44:05.879 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-29T11:44:05.879 INFO:teuthology.orchestra.run.smithi081.stderr:real 1m9.433s 2024-03-29T11:44:05.880 INFO:teuthology.orchestra.run.smithi081.stderr:user 1m57.000s 2024-03-29T11:44:05.880 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m4.245s 2024-03-29T11:44:05.880 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:44:05.880 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/remote/smithi071/log 2024-03-29T11:44:05.881 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:44:28.761 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/remote/smithi081/log 2024-03-29T11:44:28.762 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:44:46.928 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:44:47.141 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=cc77523a6af04225a67b1cb4b98d1bce Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:44:47.144 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:44:47.154 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:44:47.173 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:44:47.173 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:44:47.188 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:44:47.272 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:44:47.281 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:44:47.281 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:44:47.284 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:44:47.298 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:44:47.300 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:44:47.302 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:44:47.303 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-29T11:44:47.303 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 +418us[ +418us] +/- 70ms 2024-03-29T11:44:47.303 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -113us[ -106us] +/- 65ms 2024-03-29T11:44:47.303 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -254us[ -254us] +/- 65ms 2024-03-29T11:44:47.303 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:47.305 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:44:47.305 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-03-29T11:44:47.305 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +421us[ +421us] +/- 70ms 2024-03-29T11:44:47.305 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -121us[ -117us] +/- 65ms 2024-03-29T11:44:47.306 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -268us[ -268us] +/- 65ms 2024-03-29T11:44:47.306 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:44:47.307 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:44:47.316 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:44:47.316 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:44:47.326 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:44:47.414 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:44:47.483 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:44:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:44:47.514 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:44:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:44:47.545 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:44:47.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:44:47.566 INFO:teuthology.task.internal:Duration was 1219.097905 seconds 2024-03-29T11:44:47.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:44:47.575 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:44:47.576 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:44:47.578 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:44:47.620 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:44:47.630 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:44:48.047 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:44:48.048 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-03-29T11:44:48.048 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:44:48.067 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-03-29T11:44:48.068 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:44:48.090 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:44:48.091 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:44:48.110 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:44:48.161 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:44:48.161 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:44:48.180 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:44:48.298 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:44:48.310 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:44:48.310 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:44:48.337 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:44:48.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:44:48.376 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:44:48.380 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:44:48.403 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-03-29T11:44:48.430 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-03-29T11:44:48.476 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:44:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:44:48.490 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:44:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:44:48.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:44:48.551 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:44:48.553 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/remote/smithi071 2024-03-29T11:44:48.553 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:44:48.590 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716/remote/smithi081 2024-03-29T11:44:48.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:44:48.626 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:44:48.627 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:44:48.629 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:44:48.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:44:48.687 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:44:48.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:44:48.698 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:44:48.698 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:44:48.701 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:44:48.713 INFO:teuthology.orchestra.run.smithi071.stdout: 265566 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:44 /home/ubuntu/cephtest 2024-03-29T11:44:48.731 INFO:teuthology.orchestra.run.smithi081.stdout: 265691 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:44 /home/ubuntu/cephtest 2024-03-29T11:44:48.732 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:44:48.743 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:44:48.766 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:44:48.788 DEBUG:teuthology.nuke:shortname: smithi071 2024-03-29T11:44:48.788 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:44:48.811 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.256330', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsw3D26UudSyhWAXvfU35it+Nkk+lU6EapXFrTTx4yL3V7UtRqrKhirIsmYe1pOnNZZ/beEEY1Wztdgab8ahDU='} 2024-03-29T11:44:48.815 DEBUG:teuthology.nuke:shortname: smithi081 2024-03-29T11:44:48.816 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:44:48.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622716', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:39.257805', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPQvsr5fyqktjrh12Z71e6Rxc5GaVazbKAgCg4Zym6vhpl4LSoM7JfyI6EhK4lynfYfgcQvapZ4K0zZRCmK1gb8='} 2024-03-29T11:44:48.862 INFO:teuthology.orchestra.console.smithi071:Power off 2024-03-29T11:44:48.862 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:44:48.893 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2024-03-29T11:44:48.894 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:44:48.911 INFO:teuthology.orchestra.console.smithi081:Power off 2024-03-29T11:44:48.912 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:44:48.922 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-29T11:44:48.935 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-03-29T11:44:48.935 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:44:48.959 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-03-29T11:44:52.923 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:44:52.961 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:44:53.034 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-29T11:44:53.074 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-03-29T11:44:57.040 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:44:57.075 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:44:57.150 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-29T11:44:57.187 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-03-29T11:44:57.287 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-03-29T11:45:01.154 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:45:01.267 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2024-03-29T11:45:01.368 INFO:teuthology.orchestra.console.smithi071:Power off completed 2024-03-29T11:45:01.473 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1219.0979051589966 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=cc77523a6af04225a67b1cb4b98d1bce status: fail success: false 2024-03-29T11:45:01.473 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:45:01.547 INFO:teuthology.run:FAIL